[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:03 dist-test-slave-65w8 krb5kdc[25581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:03 dist-test-slave-65w8 krb5kdc[25581](info): set up 2 sockets
Jun 20 08:14:03 dist-test-slave-65w8 krb5kdc[25581](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 08:14:04.766201 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:04 dist-test-slave-65w8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:04.892656 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:39429
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:39429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:04.976747 25594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:04.976961 25594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:04.976986 25594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:04.978269 25594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:04.978310 25594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:04.978324 25594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:04.978338 25594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:04.978349 25594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:04.979883 25594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:39429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:39429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:04.980136 25594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:04.980389 25594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:04.987707 25600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:04.987666 25602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:04.987820 25599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:04.988741 25594 server_base.cc:1034] running on GCE node
I20260620 08:14:04.989019 25594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:04.989418 25594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:04.990608 25594 hybrid_clock.cc:648] HybridClock initialized: now 1781943244990597 us; error 62 us; skew 500 ppm
Jun 20 08:14:04 dist-test-slave-65w8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943244, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:04.993396 25594 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:04.993957 25594 webserver.cc:460] Webserver started at http://127.24.249.190:39587/ using document root <none> and password file <none>
I20260620 08:14:04.994213 25594 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:04.994288 25594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:04.994441 25594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:04.995790 25594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "4a6eb99343c94bd6a0a91840d0d869aa"
format_stamp: "Formatted at 2026-06-20 08:14:04 on dist-test-slave-65w8"
server_key: "52186d793a78708405cdf7054add24e7"
server_key_iv: "22fc90e5ac65541f2896c19c6f35e2f7"
server_key_version: "encryptionkey@0"
I20260620 08:14:04.996411 25594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "4a6eb99343c94bd6a0a91840d0d869aa"
format_stamp: "Formatted at 2026-06-20 08:14:04 on dist-test-slave-65w8"
server_key: "52186d793a78708405cdf7054add24e7"
server_key_iv: "22fc90e5ac65541f2896c19c6f35e2f7"
server_key_version: "encryptionkey@0"
I20260620 08:14:04.997903 25594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 08:14:04.998921 25609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:04.999218 25594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:04.999380 25594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "4a6eb99343c94bd6a0a91840d0d869aa"
format_stamp: "Formatted at 2026-06-20 08:14:04 on dist-test-slave-65w8"
server_key: "52186d793a78708405cdf7054add24e7"
server_key_iv: "22fc90e5ac65541f2896c19c6f35e2f7"
server_key_version: "encryptionkey@0"
I20260620 08:14:04.999493 25594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:05.019543 25594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:05.022610 25594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:05.022818 25594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:05.027590 25594 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:39429
I20260620 08:14:05.027629 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:39429 every 8 connection(s)
I20260620 08:14:05.028045 25594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:05.028460 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 25594
I20260620 08:14:05.028592 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:05.029165 25662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:05.029417 25574 external_mini_cluster.cc:1428] Setting key 7832475310525aae2fe7dd2f60f70ecd
I20260620 08:14:05.032168 25662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa: Bootstrap starting.
I20260620 08:14:05.032759 25662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:05.033113 25662 log.cc:824] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:05.033967 25662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa: No bootstrap required, opened a new log
I20260620 08:14:05.036670 25662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39429 } }
I20260620 08:14:05.036873 25662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:05.036906 25662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6eb99343c94bd6a0a91840d0d869aa, State: Initialized, Role: FOLLOWER
I20260620 08:14:05.037061 25662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39429 } }
I20260620 08:14:05.037132 25662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:05.037171 25662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:05.037220 25662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:05.037839 25662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39429 } }
I20260620 08:14:05.038014 25662 leader_election.cc:302] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a6eb99343c94bd6a0a91840d0d869aa; no voters:
I20260620 08:14:05.038218 25662 leader_election.cc:288] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:05.038295 25667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:05.038461 25667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [term 1 LEADER]: Becoming Leader. State: Replica: 4a6eb99343c94bd6a0a91840d0d869aa, State: Running, Role: LEADER
I20260620 08:14:05.038525 25662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:05.038554 25667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39429 } }
I20260620 08:14:05.038875 25667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a6eb99343c94bd6a0a91840d0d869aa. Latest consensus state: current_term: 1 leader_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39429 } } }
I20260620 08:14:05.038951 25667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:05.038893 25668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6eb99343c94bd6a0a91840d0d869aa" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39429 } } }
I20260620 08:14:05.039142 25668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:05.040488 25683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:05.040545 25683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:05.040606 25672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:05.040858 25672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:05.042701 25672 catalog_manager.cc:1347] Generated new cluster ID: 239efc49b8d94007885cbff676dca805
I20260620 08:14:05.042754 25672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:05.048885 25672 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:05.049391 25672 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:05.049463 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.030544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.24.249.190:39429)
0620 08:14:05.030749 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:05.030759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:05.031156 (+ 397us) server_negotiation.cc:407] Connection header received
0620 08:14:05.033527 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:05.033534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:05.033611 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:05.033766 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:05.035483 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.037200 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.038659 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.039291 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.039378 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.044996 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:05.045000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:05.045001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:05.045012 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:05.047103 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.047702 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.047705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.047706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.047822 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.048150 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.048151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.048151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.048428 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:05.048578 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:05.049046 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 08:14:05.049223 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260620 08:14:05.052438 25672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a6eb99343c94bd6a0a91840d0d869aa: Generated new TSK 0
I20260620 08:14:05.052606 25672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:05.085022 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39429
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:05.170580 25688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:05.170794 25688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:05.170819 25688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:05.171976 25688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:05.172022 25688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:05.172050 25688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:05.173496 25688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39429
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:05.173750 25688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:05.174024 25688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:05.179159 25693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:05.179165 25696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:05.179174 25694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:05.179427 25688 server_base.cc:1034] running on GCE node
I20260620 08:14:05.179613 25688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:05.179903 25688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:05.181057 25688 hybrid_clock.cc:648] HybridClock initialized: now 1781943245181046 us; error 28 us; skew 500 ppm
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:05.183615 25688 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:05.184028 25688 webserver.cc:460] Webserver started at http://127.24.249.129:36099/ using document root <none> and password file <none>
I20260620 08:14:05.184230 25688 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:05.184271 25688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:05.184360 25688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:05.185598 25688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "2d1045afd9de491eb1ef25f5db0d2dfe"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "00e1d79896219035b59e243710b53eb9"
server_key_iv: "6262b4ca4d35030bae32256caa22bc2a"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.186064 25688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "2d1045afd9de491eb1ef25f5db0d2dfe"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "00e1d79896219035b59e243710b53eb9"
server_key_iv: "6262b4ca4d35030bae32256caa22bc2a"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.187678 25688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:05.188541 25703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.188866 25688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:05.188959 25688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "2d1045afd9de491eb1ef25f5db0d2dfe"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "00e1d79896219035b59e243710b53eb9"
server_key_iv: "6262b4ca4d35030bae32256caa22bc2a"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.189070 25688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:05.203527 25688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:05.205955 25688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:05.206072 25688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:05.206496 25688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:05.206869 25688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:05.206910 25688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.206936 25688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:05.206950 25688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.213387 25688 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:45999
I20260620 08:14:05.213515 25816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:45999 every 8 connection(s)
I20260620 08:14:05.213850 25688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:05.221297 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 25688
I20260620 08:14:05.221413 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:05.221583 25574 external_mini_cluster.cc:1428] Setting key 2acbfdb2bc0bba1f9fb40e1d3a9f1493
I20260620 08:14:05.224249 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.214742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:53087 (local address 127.24.249.190:39429)
0620 08:14:05.214816 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:05.214819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:05.215000 (+ 181us) server_negotiation.cc:407] Connection header received
0620 08:14:05.215509 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:05.215510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:05.215520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:05.215589 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:05.216743 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.217265 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.218478 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.218739 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.218808 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.221114 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:05.221116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:05.221117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:05.221131 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:05.222754 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.223204 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.223205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.223205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.223267 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.223541 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.223542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.223542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.223675 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:05.223770 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:05.224058 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 08:14:05.224152 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:14:05.224474 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.214674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39429 (local address 127.24.249.129:53087)
0620 08:14:05.214854 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:05.214898 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:05.215385 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:05.215717 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:05.215722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:05.216392 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:05.216599 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:05.216604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.217366 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.217367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:05.218376 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:05.218379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.218844 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.218845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:05.218964 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.219489 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:05.219498 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:05.220982 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:05.222860 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:05.222868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:05.222871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:05.223109 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:05.223338 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:05.223339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:05.223339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:05.223469 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:05.223762 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:05.223768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:05.223921 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 08:14:05.224035 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:14:05.224199 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260620 08:14:05.225123 25817 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39429
I20260620 08:14:05.225240 25817 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:05.225448 25817 heartbeater.cc:511] Master 127.24.249.190:39429 requested a full tablet report, sending...
I20260620 08:14:05.226043 25626 ts_manager.cc:194] Registered new tserver with Master: 2d1045afd9de491eb1ef25f5db0d2dfe (127.24.249.129:45999)
I20260620 08:14:05.226693 25626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:53087
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:05.250175 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39429
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:05.335950 25822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:05.336169 25822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:05.336195 25822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:05.337359 25822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:05.337399 25822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:05.337427 25822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:05.339298 25822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39429
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:05.339509 25822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:05.339699 25822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:05.345410 25830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:05.345417 25828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:05.345448 25822 server_base.cc:1034] running on GCE node
W20260620 08:14:05.345417 25827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:05.346078 25822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:05.346392 25822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:05.347584 25822 hybrid_clock.cc:648] HybridClock initialized: now 1781943245347559 us; error 45 us; skew 500 ppm
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:05.350950 25822 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:05.351495 25822 webserver.cc:460] Webserver started at http://127.24.249.130:43739/ using document root <none> and password file <none>
I20260620 08:14:05.351749 25822 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:05.351821 25822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:05.351982 25822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:05.353178 25822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "0b15eefa407e441c94b3136c3c1e5cf6"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "bacd2a42f81ca7f676d6c914ee5597ca"
server_key_iv: "18df6a88c61e9e80af04e64fad1c9e37"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.353570 25822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "0b15eefa407e441c94b3136c3c1e5cf6"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "bacd2a42f81ca7f676d6c914ee5597ca"
server_key_iv: "18df6a88c61e9e80af04e64fad1c9e37"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.355315 25822 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 08:14:05.356166 25837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.356438 25822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:05.356554 25822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "0b15eefa407e441c94b3136c3c1e5cf6"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "bacd2a42f81ca7f676d6c914ee5597ca"
server_key_iv: "18df6a88c61e9e80af04e64fad1c9e37"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.356642 25822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:05.376154 25822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:05.378314 25822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:05.378480 25822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:05.378738 25822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:05.379061 25822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:05.379113 25822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.379158 25822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:05.379189 25822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.385838 25822 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:43923
I20260620 08:14:05.385891 25950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:43923 every 8 connection(s)
I20260620 08:14:05.386245 25822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:05.395918 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 25822
I20260620 08:14:05.396025 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:05.396240 25574 external_mini_cluster.cc:1428] Setting key 90e70068d2368ddc5cfce33ec47fbde0
I20260620 08:14:05.399019 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.387057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:36201 (local address 127.24.249.190:39429)
0620 08:14:05.387140 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:05.387143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:05.387311 (+ 168us) server_negotiation.cc:407] Connection header received
0620 08:14:05.388071 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:05.388073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:05.388086 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:05.388185 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:05.389439 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.390319 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.392153 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.392349 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.392457 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.395450 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:05.395452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:05.395452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:05.395460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:05.397150 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.397741 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.397743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.397743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.397810 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.398175 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.398176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.398177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.398341 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:05.398389 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:05.398828 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 08:14:05.398932 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:14:05.399546 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.386986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39429 (local address 127.24.249.130:36201)
0620 08:14:05.387157 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:05.387196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:05.387964 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:05.388301 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:05.388305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:05.389076 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:05.389345 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:05.389350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.390431 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.390433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:05.392015 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:05.392018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.392463 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.392464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:05.392611 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.393245 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:05.393255 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:05.395318 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:05.397250 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:05.397259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:05.397262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:05.397627 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:05.397923 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:05.397924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:05.397925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:05.398084 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:05.398440 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:05.398447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:05.398634 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 08:14:05.399138 (+ 504us) client_negotiation.cc:240] Negotiation successful
0620 08:14:05.399333 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260620 08:14:05.400199 25951 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39429
I20260620 08:14:05.400326 25951 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:05.400517 25951 heartbeater.cc:511] Master 127.24.249.190:39429 requested a full tablet report, sending...
I20260620 08:14:05.400933 25626 ts_manager.cc:194] Registered new tserver with Master: 0b15eefa407e441c94b3136c3c1e5cf6 (127.24.249.130:43923)
I20260620 08:14:05.401329 25626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:36201
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:05.427259 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39429
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:05.515691 25956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:05.515894 25956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:05.515918 25956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:05.517148 25956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:05.517195 25956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:05.517221 25956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:05.518740 25956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39429
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:05.518932 25956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:05.519109 25956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:05.524281 25961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:05.524298 25962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:05.524539 25964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:05.524533 25956 server_base.cc:1034] running on GCE node
I20260620 08:14:05.524798 25956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:05.525034 25956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:05.526187 25956 hybrid_clock.cc:648] HybridClock initialized: now 1781943245526139 us; error 61 us; skew 500 ppm
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:05.528987 25956 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:05.529467 25956 webserver.cc:460] Webserver started at http://127.24.249.131:39933/ using document root <none> and password file <none>
I20260620 08:14:05.529721 25956 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:05.529786 25956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:05.529975 25956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:05.531067 25956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "c53ad81888e34abf8471ef9e8e1d2efd"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "f4d4d08afac3eae1ff33f3598fca55f7"
server_key_iv: "a9422eae203a924e35765d15b060b34d"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.531440 25956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "c53ad81888e34abf8471ef9e8e1d2efd"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "f4d4d08afac3eae1ff33f3598fca55f7"
server_key_iv: "a9422eae203a924e35765d15b060b34d"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.532837 25956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:05.533752 25971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.533977 25956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:05.534083 25956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "c53ad81888e34abf8471ef9e8e1d2efd"
format_stamp: "Formatted at 2026-06-20 08:14:05 on dist-test-slave-65w8"
server_key: "f4d4d08afac3eae1ff33f3598fca55f7"
server_key_iv: "a9422eae203a924e35765d15b060b34d"
server_key_version: "encryptionkey@0"
I20260620 08:14:05.534173 25956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:05.553596 25956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:05.555795 25956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:05.555956 25956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:05.556186 25956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:05.556502 25956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:05.556551 25956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.556593 25956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:05.556622 25956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:05.563088 25956 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:33277
I20260620 08:14:05.563128 26084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:33277 every 8 connection(s)
I20260620 08:14:05.563450 25956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:05.572659 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.564143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:48497 (local address 127.24.249.190:39429)
0620 08:14:05.564188 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:05.564190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:05.564377 (+ 187us) server_negotiation.cc:407] Connection header received
0620 08:14:05.564842 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:05.564842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:05.564853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:05.564900 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:05.565921 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.566493 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.567599 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.567786 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.567847 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.570210 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:05.570212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:05.570212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:05.570239 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:05.571454 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.571844 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.571844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.571845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.571896 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.572110 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.572110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.572110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.572215 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:05.572285 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:05.572530 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 08:14:05.572592 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:05.572930 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.564124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39429 (local address 127.24.249.131:48497)
0620 08:14:05.564259 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:05.564294 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:05.564762 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:05.564963 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:05.564967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:05.565542 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:05.565768 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:05.565773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.566599 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.566600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:05.567503 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:05.567505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.567869 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.567870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:05.567986 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.568612 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:05.568620 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:05.570105 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:05.571527 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:05.571533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:05.571535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:05.571775 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:05.571957 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:05.571957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:05.571958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:05.572057 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:05.572290 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:05.572295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:05.572415 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 08:14:05.572506 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 08:14:05.572627 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260620 08:14:05.573042 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 25956
I20260620 08:14:05.573112 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:05.573308 25574 external_mini_cluster.cc:1428] Setting key defefaa0d0e9c0cbd519d973a5e07fdd
I20260620 08:14:05.573441 26085 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39429
I20260620 08:14:05.573541 26085 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:05.573767 26085 heartbeater.cc:511] Master 127.24.249.190:39429 requested a full tablet report, sending...
I20260620 08:14:05.574204 25625 ts_manager.cc:194] Registered new tserver with Master: c53ad81888e34abf8471ef9e8e1d2efd (127.24.249.131:33277)
I20260620 08:14:05.574682 25625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:48497
I20260620 08:14:05.585554 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:05.610067 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.601469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.24.249.129:45999)
0620 08:14:05.601657 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:05.601662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:05.601673 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:05.601725 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:05.601728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:05.601742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:05.601818 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:05.602369 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.603119 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.603890 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.604210 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.604274 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.607009 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:05.607017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:05.607017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:05.607024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:05.608514 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.609038 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.609041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.609041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.609102 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.609337 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.609338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.609338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.609497 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:05.609699 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:05.609739 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 08:14:05.609827 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
W20260620 08:14:05.610601 25731 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34126
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[25581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:05.634728 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:05.626889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.24.249.129:45999)
0620 08:14:05.626982 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:05.626984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:05.626997 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:14:05.627070 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:05.627071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:05.627086 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:05.627157 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:05.627757 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.628321 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.629150 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:05.629386 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:05.629451 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:05.631879 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:05.631880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:05.631880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:05.631887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:05.633310 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.633713 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.633714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.633715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.633786 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:05.634062 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:05.634063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:05.634063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:05.634175 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:05.634277 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:05.634452 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:14:05.634534 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 08:14:05.635519 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 25688
I20260620 08:14:05.642467 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 25822
I20260620 08:14:05.648129 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 25956
I20260620 08:14:05.652694 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 25594
2026-06-20T08:14:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1989 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[26099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[26099](info): set up 2 sockets
Jun 20 08:14:05 dist-test-slave-65w8 krb5kdc[26099](info): commencing operation
krb5kdc: starting...
W20260620 08:14:06.721554 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:06 dist-test-slave-65w8 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:06.842792 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40425
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:40425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:06.922901 26112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:06.923105 26112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:06.923127 26112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:06.924232 26112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:06.924275 26112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:06.924289 26112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:06.924302 26112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:06.924314 26112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:06.925697 26112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:40425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:06.925952 26112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:06.926146 26112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:06.931172 26120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:06.931354 26112 server_base.cc:1034] running on GCE node
W20260620 08:14:06.931161 26117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:06.931208 26118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:06.931684 26112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:06.931998 26112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:06.933172 26112 hybrid_clock.cc:648] HybridClock initialized: now 1781943246933150 us; error 47 us; skew 500 ppm
Jun 20 08:14:06 dist-test-slave-65w8 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943246, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:06.935904 26112 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:06.936363 26112 webserver.cc:460] Webserver started at http://127.24.249.190:46609/ using document root <none> and password file <none>
I20260620 08:14:06.936605 26112 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:06.936671 26112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:06.936815 26112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:06.937952 26112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "66a05c14ae6a499c9b5343f6abf4b57f"
format_stamp: "Formatted at 2026-06-20 08:14:06 on dist-test-slave-65w8"
server_key: "20767be12aadf20cab7bf5a72273359f"
server_key_iv: "bdfccaf33f866403f8d7f1f409622671"
server_key_version: "encryptionkey@0"
I20260620 08:14:06.938318 26112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "66a05c14ae6a499c9b5343f6abf4b57f"
format_stamp: "Formatted at 2026-06-20 08:14:06 on dist-test-slave-65w8"
server_key: "20767be12aadf20cab7bf5a72273359f"
server_key_iv: "bdfccaf33f866403f8d7f1f409622671"
server_key_version: "encryptionkey@0"
I20260620 08:14:06.939639 26112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:06.940424 26127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:06.940621 26112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:06.940718 26112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "66a05c14ae6a499c9b5343f6abf4b57f"
format_stamp: "Formatted at 2026-06-20 08:14:06 on dist-test-slave-65w8"
server_key: "20767be12aadf20cab7bf5a72273359f"
server_key_iv: "bdfccaf33f866403f8d7f1f409622671"
server_key_version: "encryptionkey@0"
I20260620 08:14:06.940809 26112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:06.952816 26112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:06.955114 26112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:06.955281 26112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:06.959239 26112 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:40425
I20260620 08:14:06.959293 26179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:40425 every 8 connection(s)
I20260620 08:14:06.959645 26112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:06.960541 26180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:06.962899 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f: Bootstrap starting.
I20260620 08:14:06.963474 26180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:06.963774 26180 log.cc:824] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:06.964443 26180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f: No bootstrap required, opened a new log
I20260620 08:14:06.966079 26180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40425 } }
I20260620 08:14:06.966214 26180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:06.966236 26180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a05c14ae6a499c9b5343f6abf4b57f, State: Initialized, Role: FOLLOWER
I20260620 08:14:06.966400 26180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40425 } }
I20260620 08:14:06.966471 26180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:06.966492 26180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:06.966559 26180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:06.967083 26180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40425 } }
I20260620 08:14:06.967167 26180 leader_election.cc:302] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a05c14ae6a499c9b5343f6abf4b57f; no voters:
I20260620 08:14:06.967342 26180 leader_election.cc:288] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:06.967607 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 26112
I20260620 08:14:06.967674 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:06.967859 25574 external_mini_cluster.cc:1428] Setting key 0a5c51cb0087d8268151df8d08591fb5
I20260620 08:14:06.967397 26183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:06.967618 26180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:06.968158 26183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [term 1 LEADER]: Becoming Leader. State: Replica: 66a05c14ae6a499c9b5343f6abf4b57f, State: Running, Role: LEADER
I20260620 08:14:06.968262 26183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40425 } }
I20260620 08:14:06.968948 26184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40425 } } }
I20260620 08:14:06.969044 26184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:06.969257 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a05c14ae6a499c9b5343f6abf4b57f. Latest consensus state: current_term: 1 leader_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a05c14ae6a499c9b5343f6abf4b57f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40425 } } }
I20260620 08:14:06.969398 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:06.970044 26201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:06.970122 26201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:06.970165 26195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:06.970311 26195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:06.971833 26195 catalog_manager.cc:1347] Generated new cluster ID: 6717d84992894ba3b01fb42d7781049b
I20260620 08:14:06.971884 26195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:14:06 dist-test-slave-65w8 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:06.980507 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:06.968514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.24.249.190:40425)
0620 08:14:06.968668 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:06.968677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:06.968869 (+ 192us) server_negotiation.cc:407] Connection header received
0620 08:14:06.969080 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:06.969088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:06.969171 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:06.969290 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:06.970208 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:06.971420 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:06.972516 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:06.973106 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:06.973201 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:06.975989 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:06.975993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:06.975993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:06.976005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:06.978768 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:06.979213 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:06.979216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:06.979217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:06.979342 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:06.979548 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:06.979549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:06.979549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:06.979865 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:06.980035 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:06.980147 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:14:06.980291 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260620 08:14:06.989264 26195 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:06.989989 26195 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:06.996711 26195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66a05c14ae6a499c9b5343f6abf4b57f: Generated new TSK 0
I20260620 08:14:06.996945 26195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:07.030495 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40425
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:07.125537 26206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:07.125795 26206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:07.125850 26206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:07.127054 26206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:07.127094 26206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:07.127121 26206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:07.129277 26206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:07.129549 26206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:07.129833 26206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:07.136319 26212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:07.136467 26214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:07.136490 26206 server_base.cc:1034] running on GCE node
W20260620 08:14:07.136319 26211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:07.136896 26206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:07.137176 26206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:07.138345 26206 hybrid_clock.cc:648] HybridClock initialized: now 1781943247138327 us; error 32 us; skew 500 ppm
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:07.141412 26206 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:07.142017 26206 webserver.cc:460] Webserver started at http://127.24.249.129:45557/ using document root <none> and password file <none>
I20260620 08:14:07.142302 26206 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:07.142364 26206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:07.142511 26206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:07.143977 26206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "930cc67c45004194a620d15d3e7aa44c"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "86cbf052e034681caf6f81e4754e2b47"
server_key_iv: "01914ff264498315bf4948d40dda7cae"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.144403 26206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "930cc67c45004194a620d15d3e7aa44c"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "86cbf052e034681caf6f81e4754e2b47"
server_key_iv: "01914ff264498315bf4948d40dda7cae"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.146347 26206 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 08:14:07.147306 26221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.147475 26206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:07.147568 26206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "930cc67c45004194a620d15d3e7aa44c"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "86cbf052e034681caf6f81e4754e2b47"
server_key_iv: "01914ff264498315bf4948d40dda7cae"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.147665 26206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:07.161501 26206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:07.163534 26206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:07.163666 26206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:07.163847 26206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:07.164170 26206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:07.164206 26206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.164251 26206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:07.164275 26206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.170732 26206 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:40591
I20260620 08:14:07.170814 26334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:40591 every 8 connection(s)
I20260620 08:14:07.171129 26206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:07.176018 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 26206
I20260620 08:14:07.176142 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:07.176326 25574 external_mini_cluster.cc:1428] Setting key ace1da78ca1e42368545abce5f64016d
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:07.181747 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.171903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:39565 (local address 127.24.249.190:40425)
0620 08:14:07.171976 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.171978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.172141 (+ 163us) server_negotiation.cc:407] Connection header received
0620 08:14:07.173152 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.173153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.173170 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.173242 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:07.173852 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.174492 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.175559 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.176079 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.176177 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.178462 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:07.178463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:07.178464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:07.178474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:07.180044 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.180569 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.180570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.180570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.180697 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.181026 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.181027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.181027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.181140 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:07.181217 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.181561 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.181647 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:14:07.182034 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.171860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40425 (local address 127.24.249.129:39565)
0620 08:14:07.172017 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.172059 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.173030 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.173311 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.173315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.173562 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:07.173774 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.173779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.174612 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.174613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.175463 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.175466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.176194 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.176196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.176335 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.176922 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:07.176931 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:07.178349 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:07.180128 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:07.180135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:07.180137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:07.180479 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:07.180778 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:07.180780 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:07.180780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:07.180933 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:07.181226 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:07.181233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:07.181406 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.181528 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.181719 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260620 08:14:07.182603 26335 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40425
I20260620 08:14:07.182714 26335 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:07.182911 26335 heartbeater.cc:511] Master 127.24.249.190:40425 requested a full tablet report, sending...
I20260620 08:14:07.183393 26144 ts_manager.cc:194] Registered new tserver with Master: 930cc67c45004194a620d15d3e7aa44c (127.24.249.129:40591)
I20260620 08:14:07.184041 26144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:39565
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:07.205037 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40425
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:07.287360 26340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:07.287572 26340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:07.287596 26340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:07.288774 26340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:07.288818 26340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:07.288846 26340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:07.290391 26340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:07.290639 26340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:07.290880 26340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:07.296020 26345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:07.296105 26346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:07.296121 26348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:07.296118 26340 server_base.cc:1034] running on GCE node
I20260620 08:14:07.296633 26340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:07.296900 26340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:07.298066 26340 hybrid_clock.cc:648] HybridClock initialized: now 1781943247298050 us; error 38 us; skew 500 ppm
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:07.300665 26340 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:07.301177 26340 webserver.cc:460] Webserver started at http://127.24.249.130:46187/ using document root <none> and password file <none>
I20260620 08:14:07.301421 26340 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:07.301491 26340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:07.301676 26340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:07.302889 26340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "c8c023ec6c4a4fd593dff40bff874468"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "4db49dbc838df22fb27a2f1b638ab20d"
server_key_iv: "b7d6737983ac4a9a0898375d379c271c"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.303285 26340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "c8c023ec6c4a4fd593dff40bff874468"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "4db49dbc838df22fb27a2f1b638ab20d"
server_key_iv: "b7d6737983ac4a9a0898375d379c271c"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.304873 26340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:07.305588 26355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.305770 26340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:07.305910 26340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "c8c023ec6c4a4fd593dff40bff874468"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "4db49dbc838df22fb27a2f1b638ab20d"
server_key_iv: "b7d6737983ac4a9a0898375d379c271c"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.306002 26340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:07.318091 26340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:07.320546 26340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:07.320725 26340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:07.320976 26340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:07.321306 26340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:07.321362 26340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.321408 26340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:07.321435 26340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.327960 26340 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:37493
I20260620 08:14:07.328003 26468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:37493 every 8 connection(s)
I20260620 08:14:07.328347 26340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:07.330426 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 26340
I20260620 08:14:07.330528 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:07.330756 25574 external_mini_cluster.cc:1428] Setting key 679eb796a9a7d8059850053149a09827
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:14:07.339799 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.329076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:55731 (local address 127.24.249.190:40425)
0620 08:14:07.329123 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.329125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.329374 (+ 249us) server_negotiation.cc:407] Connection header received
0620 08:14:07.330465 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.330467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.330481 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.330568 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:07.331492 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.332003 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.333575 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.333926 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.334020 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.336657 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:07.336659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:07.336659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:07.336667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:07.338323 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.338755 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.338757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.338757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.338850 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.339115 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.339116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.339117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.339232 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:07.339313 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.339608 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.339695 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:07.340001 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.329040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40425 (local address 127.24.249.130:55731)
0620 08:14:07.329219 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.329265 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.330332 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.330724 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.330730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.331117 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:07.331409 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.331415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.332097 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.332098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.333469 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.333472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.334035 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.334036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.334183 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.334734 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:07.334745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:07.336543 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:07.338425 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:07.338432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:07.338435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:07.338673 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:07.338919 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:07.338919 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:07.338920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:07.339041 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:07.339303 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:07.339310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:07.339478 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.339588 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.339751 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260620 08:14:07.340559 26469 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40425
I20260620 08:14:07.340646 26469 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:07.340843 26469 heartbeater.cc:511] Master 127.24.249.190:40425 requested a full tablet report, sending...
I20260620 08:14:07.341318 26144 ts_manager.cc:194] Registered new tserver with Master: c8c023ec6c4a4fd593dff40bff874468 (127.24.249.130:37493)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
I20260620 08:14:07.341882 26144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:55731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:07.357445 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40425
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:07.441777 26474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:07.442065 26474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:07.442104 26474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:07.443303 26474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:07.443382 26474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:07.443427 26474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:07.445091 26474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:07.445312 26474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:07.445497 26474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:07.450685 26482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:07.450791 26479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:07.450896 26480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:07.451004 26474 server_base.cc:1034] running on GCE node
I20260620 08:14:07.451295 26474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:07.451551 26474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:07.452755 26474 hybrid_clock.cc:648] HybridClock initialized: now 1781943247452732 us; error 37 us; skew 500 ppm
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:07.456144 26474 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:07.456634 26474 webserver.cc:460] Webserver started at http://127.24.249.131:35877/ using document root <none> and password file <none>
I20260620 08:14:07.456908 26474 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:07.456976 26474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:07.457118 26474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:07.458282 26474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "e29a279ca155415bbd7168690e024290"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "e7adb353a88aae36849f88736d473bec"
server_key_iv: "f0a3aee2dfbd5b8ea285e8a358ac480a"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.458634 26474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "e29a279ca155415bbd7168690e024290"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "e7adb353a88aae36849f88736d473bec"
server_key_iv: "f0a3aee2dfbd5b8ea285e8a358ac480a"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.459990 26474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:07.460701 26489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.460976 26474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:07.461095 26474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "e29a279ca155415bbd7168690e024290"
format_stamp: "Formatted at 2026-06-20 08:14:07 on dist-test-slave-65w8"
server_key: "e7adb353a88aae36849f88736d473bec"
server_key_iv: "f0a3aee2dfbd5b8ea285e8a358ac480a"
server_key_version: "encryptionkey@0"
I20260620 08:14:07.461169 26474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:07.481925 26474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:07.484477 26474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:07.484660 26474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:07.484915 26474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:07.485234 26474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:07.485288 26474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.485329 26474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:07.485361 26474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:07.491994 26474 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:46853
I20260620 08:14:07.492038 26602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:46853 every 8 connection(s)
I20260620 08:14:07.492416 26474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:07.492913 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 26474
I20260620 08:14:07.493023 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:07.493180 25574 external_mini_cluster.cc:1428] Setting key cd87997982a0841caeb5a259476d11c6
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:07.502964 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.493365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:40847 (local address 127.24.249.190:40425)
0620 08:14:07.493431 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.493433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.493664 (+ 231us) server_negotiation.cc:407] Connection header received
0620 08:14:07.494803 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.494804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.494815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.494873 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:07.495511 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.496102 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.497363 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.497569 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.497642 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.500073 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:07.500074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:07.500075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:07.500081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:07.501608 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.502076 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.502077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.502078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.502133 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.502409 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.502410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.502410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.502514 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:07.502564 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.502832 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.502894 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:07.503192 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.493288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40425 (local address 127.24.249.131:40847)
0620 08:14:07.493540 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.493582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.494701 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.494945 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.494948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.495211 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:07.495403 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.495408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.496208 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.496210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.497274 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.497277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.497796 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.497798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.498005 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.498533 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:07.498542 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:07.499949 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:07.501691 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:07.501698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:07.501700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:07.502003 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:07.502225 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:07.502226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:07.502226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:07.502341 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:07.502564 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:07.502569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:07.502718 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.502800 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.502928 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260620 08:14:07.503713 26603 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40425
I20260620 08:14:07.503878 26603 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:07.504072 26603 heartbeater.cc:511] Master 127.24.249.190:40425 requested a full tablet report, sending...
I20260620 08:14:07.504475 26144 ts_manager.cc:194] Registered new tserver with Master: e29a279ca155415bbd7168690e024290 (127.24.249.131:46853)
I20260620 08:14:07.504865 26144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:40847
I20260620 08:14:07.504911 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:14:07.513089 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.506608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.24.249.190:40425)
0620 08:14:07.506662 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.506665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.506783 (+ 118us) server_negotiation.cc:407] Connection header received
0620 08:14:07.506826 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.506827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.506835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.506881 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:07.507395 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.507886 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.508713 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.508949 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.509012 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.510382 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:07.510383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:07.510384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:07.510390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:07.511944 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.512306 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.512307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.512307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.512362 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.512530 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.512530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.512530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.512672 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:07.512781 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.512932 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.512995 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:14:07.515295 26144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:07.517308 26144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:07.525095 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.520716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.24.249.131:46853)
0620 08:14:07.521311 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.521317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.521327 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:07.521362 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.521364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.521378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.521447 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:07.521617 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.522760 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.523961 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.524664 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.524758 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.524818 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.524964 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.525005 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":242,"threads_started":1}
I20260620 08:14:07.525113 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.520469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46853 (local address 127.0.0.1:40398)
0620 08:14:07.520876 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.520893 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.520975 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.521483 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.521484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.521486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:07.521549 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.521550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.522891 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.522892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.523871 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.523873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.524756 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.524758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.524839 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.524855 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.524941 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.524998 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260620 08:14:07.525345 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.520475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:40591 (local address 127.0.0.1:42408)
0620 08:14:07.521114 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.521124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.521174 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.521391 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.521394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.521397 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:07.521477 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.521479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.522868 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.522869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.524073 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.524076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.525064 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.525065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.525141 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.525146 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.525198 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.525242 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":4,"thread_start_us":47,"threads_started":1}
I20260620 08:14:07.525540 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.521087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.24.249.130:37493)
0620 08:14:07.521595 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.521600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.521610 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:07.521647 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.521649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.521662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.521755 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:07.521869 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.522546 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.524312 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.524884 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.524951 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.525001 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.525426 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.525471 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":99,"threads_started":1}
I20260620 08:14:07.525592 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.520587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.24.249.129:40591)
0620 08:14:07.520973 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.520978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.521221 (+ 243us) server_negotiation.cc:407] Connection header received
0620 08:14:07.521275 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.521277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.521295 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.521379 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:07.521564 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.522731 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.524164 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.524884 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.524978 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.525029 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.525464 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.525508 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260620 08:14:07.525718 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.520469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:37493 (local address 127.0.0.1:47864)
0620 08:14:07.520733 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.520936 (+ 203us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.521011 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.521752 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.521753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.521755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:07.521802 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.521804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.522665 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.522667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.524228 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.524230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.525217 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.525217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.525269 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.525272 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.525320 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.525356 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":56,"thread_start_us":95,"threads_started":1}
I20260620 08:14:07.526433 26537 tablet_service.cc:1467] Processing CreateTablet for tablet 4207f6d82ee449d18551657f9c195dbf (DEFAULT_TABLE table=test-table [id=1e4b9029976c4ac9b6a197890dd265d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:07.526439 26403 tablet_service.cc:1467] Processing CreateTablet for tablet 4207f6d82ee449d18551657f9c195dbf (DEFAULT_TABLE table=test-table [id=1e4b9029976c4ac9b6a197890dd265d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:07.526763 26403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4207f6d82ee449d18551657f9c195dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:07.526765 26537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4207f6d82ee449d18551657f9c195dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:07.529469 26620 tablet_bootstrap.cc:492] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290: Bootstrap starting.
I20260620 08:14:07.529831 26621 tablet_bootstrap.cc:492] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468: Bootstrap starting.
I20260620 08:14:07.530248 26620 tablet_bootstrap.cc:654] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:07.530151 26269 tablet_service.cc:1467] Processing CreateTablet for tablet 4207f6d82ee449d18551657f9c195dbf (DEFAULT_TABLE table=test-table [id=1e4b9029976c4ac9b6a197890dd265d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:07.530508 26621 tablet_bootstrap.cc:654] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:07.530524 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4207f6d82ee449d18551657f9c195dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:07.530853 26620 log.cc:824] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:07.531073 26621 log.cc:824] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:07.531610 26620 tablet_bootstrap.cc:492] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290: No bootstrap required, opened a new log
I20260620 08:14:07.531719 26620 ts_tablet_manager.cc:1397] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:14:07.531857 26621 tablet_bootstrap.cc:492] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468: No bootstrap required, opened a new log
I20260620 08:14:07.531919 26621 ts_tablet_manager.cc:1397] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:14:07.533654 26624 tablet_bootstrap.cc:492] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c: Bootstrap starting.
I20260620 08:14:07.534255 26620 raft_consensus.cc:348] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.534463 26620 raft_consensus.cc:374] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:07.534509 26620 raft_consensus.cc:729] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29a279ca155415bbd7168690e024290, State: Initialized, Role: FOLLOWER
I20260620 08:14:07.534405 26621 raft_consensus.cc:348] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.534550 26621 raft_consensus.cc:374] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:07.534579 26621 raft_consensus.cc:729] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c023ec6c4a4fd593dff40bff874468, State: Initialized, Role: FOLLOWER
I20260620 08:14:07.534598 26624 tablet_bootstrap.cc:654] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:07.534649 26620 consensus_queue.cc:260] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.534696 26621 consensus_queue.cc:260] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.534974 26621 ts_tablet_manager.cc:1428] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 08:14:07.534974 26620 ts_tablet_manager.cc:1428] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260620 08:14:07.535131 26603 heartbeater.cc:503] Master 127.24.249.190:40425 was elected leader, sending a full tablet report...
I20260620 08:14:07.535178 26624 log.cc:824] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:07.535444 26469 heartbeater.cc:503] Master 127.24.249.190:40425 was elected leader, sending a full tablet report...
I20260620 08:14:07.536187 26624 tablet_bootstrap.cc:492] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c: No bootstrap required, opened a new log
I20260620 08:14:07.536276 26624 ts_tablet_manager.cc:1397] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:14:07.537850 26624 raft_consensus.cc:348] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.538067 26624 raft_consensus.cc:374] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:07.538091 26624 raft_consensus.cc:729] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930cc67c45004194a620d15d3e7aa44c, State: Initialized, Role: FOLLOWER
I20260620 08:14:07.538201 26624 consensus_queue.cc:260] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.538528 26335 heartbeater.cc:503] Master 127.24.249.190:40425 was elected leader, sending a full tablet report...
I20260620 08:14:07.538518 26624 ts_tablet_manager.cc:1428] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260620 08:14:07.579032 26470 tablet.cc:2369] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:07.658687 26628 raft_consensus.cc:482] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:07.658824 26628 raft_consensus.cc:504] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.659293 26628 leader_election.cc:288] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e29a279ca155415bbd7168690e024290 (127.24.249.131:46853), c8c023ec6c4a4fd593dff40bff874468 (127.24.249.130:37493)
I20260620 08:14:07.663264 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.659545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:38349 (local address 127.24.249.131:46853)
0620 08:14:07.659588 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.659590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.659721 (+ 131us) server_negotiation.cc:407] Connection header received
0620 08:14:07.659750 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.659751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.659761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.659814 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:07.659953 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.660688 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.662271 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.662872 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.662954 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.662964 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.663137 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.663193 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:07.663265 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.659545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:36679 (local address 127.24.249.130:37493)
0620 08:14:07.659601 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.659603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.659615 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:07.659671 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.659672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.659680 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.659764 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:07.659941 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.660724 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.662277 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.662967 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.663033 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.663039 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.663175 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.663209 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:07.663268 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.659476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:37493 (local address 127.24.249.129:36679)
0620 08:14:07.659540 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.659553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.659601 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.659777 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.659777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.659779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:07.659853 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.659856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.660799 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.660800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.662165 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.662168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.663038 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.663039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.663111 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.663114 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.663165 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.663195 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":57}
I20260620 08:14:07.663273 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.659485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46853 (local address 127.24.249.129:38349)
0620 08:14:07.659645 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:07.659654 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:07.659709 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:07.659821 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:07.659822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:07.659823 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:07.659888 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.659889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.660819 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.660819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.662165 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:07.662168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.662963 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.662964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:07.663058 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.663063 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:07.663126 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:14:07.663176 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":98,"thread_start_us":60,"threads_started":1}
I20260620 08:14:07.663748 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4207f6d82ee449d18551657f9c195dbf" candidate_uuid: "930cc67c45004194a620d15d3e7aa44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29a279ca155415bbd7168690e024290" is_pre_election: true
I20260620 08:14:07.663883 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4207f6d82ee449d18551657f9c195dbf" candidate_uuid: "930cc67c45004194a620d15d3e7aa44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c023ec6c4a4fd593dff40bff874468" is_pre_election: true
I20260620 08:14:07.663920 26557 raft_consensus.cc:2454] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930cc67c45004194a620d15d3e7aa44c in term 0.
I20260620 08:14:07.664007 26423 raft_consensus.cc:2454] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930cc67c45004194a620d15d3e7aa44c in term 0.
I20260620 08:14:07.664143 26225 leader_election.cc:302] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930cc67c45004194a620d15d3e7aa44c, e29a279ca155415bbd7168690e024290; no voters:
I20260620 08:14:07.664273 26628 raft_consensus.cc:2781] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:07.664353 26628 raft_consensus.cc:482] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:07.664391 26628 raft_consensus.cc:3037] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:07.665138 26628 raft_consensus.cc:504] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.665280 26628 leader_election.cc:288] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [CANDIDATE]: Term 1 election: Requested vote from peers e29a279ca155415bbd7168690e024290 (127.24.249.131:46853), c8c023ec6c4a4fd593dff40bff874468 (127.24.249.130:37493)
I20260620 08:14:07.665443 26423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4207f6d82ee449d18551657f9c195dbf" candidate_uuid: "930cc67c45004194a620d15d3e7aa44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8c023ec6c4a4fd593dff40bff874468"
I20260620 08:14:07.665477 26557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4207f6d82ee449d18551657f9c195dbf" candidate_uuid: "930cc67c45004194a620d15d3e7aa44c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29a279ca155415bbd7168690e024290"
I20260620 08:14:07.665537 26423 raft_consensus.cc:3037] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:07.665536 26557 raft_consensus.cc:3037] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:07.666294 26557 raft_consensus.cc:2454] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930cc67c45004194a620d15d3e7aa44c in term 1.
I20260620 08:14:07.666299 26423 raft_consensus.cc:2454] T 4207f6d82ee449d18551657f9c195dbf P c8c023ec6c4a4fd593dff40bff874468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930cc67c45004194a620d15d3e7aa44c in term 1.
I20260620 08:14:07.666458 26225 leader_election.cc:302] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930cc67c45004194a620d15d3e7aa44c, e29a279ca155415bbd7168690e024290; no voters:
I20260620 08:14:07.666543 26628 raft_consensus.cc:2781] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:07.666684 26628 raft_consensus.cc:686] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [term 1 LEADER]: Becoming Leader. State: Replica: 930cc67c45004194a620d15d3e7aa44c, State: Running, Role: LEADER
I20260620 08:14:07.666759 26628 consensus_queue.cc:237] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } }
I20260620 08:14:07.667701 26143 catalog_manager.cc:5495] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c reported cstate change: term changed from 0 to 1, leader changed from <none> to 930cc67c45004194a620d15d3e7aa44c (127.24.249.129). New cstate: current_term: 1 leader_uuid: "930cc67c45004194a620d15d3e7aa44c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930cc67c45004194a620d15d3e7aa44c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e29a279ca155415bbd7168690e024290" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8c023ec6c4a4fd593dff40bff874468" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 37493 } health_report { overall_health: UNKNOWN } } }
W20260620 08:14:07.671875 26336 tablet.cc:2369] T 4207f6d82ee449d18551657f9c195dbf P 930cc67c45004194a620d15d3e7aa44c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:07.713654 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.705494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.24.249.129:40591)
0620 08:14:07.705564 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.705567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.705676 (+ 109us) server_negotiation.cc:407] Connection header received
0620 08:14:07.705705 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.705706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.705717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.705808 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:07.706421 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.707006 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.707807 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.707996 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.708068 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.710825 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:07.710836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:07.710836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:07.710843 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:07.712451 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.712815 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.712818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.712818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.712872 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.713121 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.713122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.713122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.713285 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:07.713435 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.713511 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.713580 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:07.738788 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.730655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.24.249.129:40591)
0620 08:14:07.730744 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.730746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.730836 (+ 90us) server_negotiation.cc:407] Connection header received
0620 08:14:07.730939 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.730940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.730951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.731025 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:07.731684 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.732257 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.733032 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.733242 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.733328 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.736161 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:07.736162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:07.736163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:07.736170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:07.737580 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.738002 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.738003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.738003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.738070 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.738282 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.738282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.738283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.738391 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:07.738471 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.738626 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.738713 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 08:14:07.743077 26604 tablet.cc:2369] T 4207f6d82ee449d18551657f9c195dbf P e29a279ca155415bbd7168690e024290: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:07.762142 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:07.754565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.24.249.129:40591)
0620 08:14:07.754625 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:07.754627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:07.754666 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:14:07.754737 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:07.754738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:07.754747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:07.754797 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:07.755335 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.755835 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.756590 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:07.756807 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:07.756867 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:07.759339 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:07.759341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:07.759342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:07.759350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:07.760892 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.761289 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.761290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.761291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.761348 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:07.761557 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:07.761558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:07.761558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:07.761672 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:07.761782 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:07.761932 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 08:14:07.762015 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 08:14:07.762669 26249 tablet_service.cc:2908] Rejecting scan request for tablet 4207f6d82ee449d18551657f9c195dbf: Uninitialized: clean time has not yet been initialized
I20260620 08:14:07.763593 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 26206
I20260620 08:14:07.769361 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 26340
I20260620 08:14:07.774096 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 26474
I20260620 08:14:07.778499 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 26112
2026-06-20T08:14:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2126 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26644](info): set up 2 sockets
Jun 20 08:14:07 dist-test-slave-65w8 krb5kdc[26644](info): commencing operation
krb5kdc: starting...
W20260620 08:14:08.841686 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:08 dist-test-slave-65w8 krb5kdc[26644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:08.960150 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:45401
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:45401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:09.043561 26657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:09.043802 26657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:09.043825 26657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:09.044953 26657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:09.044988 26657 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:14:09.045001 26657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:09.045014 26657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:09.045027 26657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:09.045039 26657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:09.046561 26657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:45401
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:45401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:09.046772 26657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:09.046948 26657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:09.051726 26665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.051726 26662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.051746 26663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:09.051909 26657 server_base.cc:1034] running on GCE node
I20260620 08:14:09.052124 26657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:09.052425 26657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:09.053588 26657 hybrid_clock.cc:648] HybridClock initialized: now 1781943249053564 us; error 40 us; skew 500 ppm
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:09.061506 26657 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:09.062108 26657 webserver.cc:460] Webserver started at http://127.24.249.190:37305/ using document root <none> and password file <none>
I20260620 08:14:09.062369 26657 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:09.062438 26657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:09.062600 26657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:09.063723 26657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "791985de669f4265a2faf2dfce498f4c"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "99da8f684fd22ef7d9f9a9cdb42c06a2"
server_key_iv: "d9a6dbe82be7027429641606e06d5b9e"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.064107 26657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "791985de669f4265a2faf2dfce498f4c"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "99da8f684fd22ef7d9f9a9cdb42c06a2"
server_key_iv: "d9a6dbe82be7027429641606e06d5b9e"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.065522 26657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:09.066495 26672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.066709 26657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:09.066828 26657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "791985de669f4265a2faf2dfce498f4c"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "99da8f684fd22ef7d9f9a9cdb42c06a2"
server_key_iv: "d9a6dbe82be7027429641606e06d5b9e"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.066905 26657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:09.081835 26657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:09.084194 26657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:09.084372 26657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:09.089465 26657 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:45401
I20260620 08:14:09.089511 26734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:45401 every 8 connection(s)
I20260620 08:14:09.089898 26657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:09.090564 26735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:09.092744 26735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c: Bootstrap starting.
I20260620 08:14:09.093290 26735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:09.093526 26735 log.cc:824] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:09.094309 26735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c: No bootstrap required, opened a new log
I20260620 08:14:09.095458 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 26657
I20260620 08:14:09.095568 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:09.095731 25574 external_mini_cluster.cc:1428] Setting key b3f0a54265f804ddf3d383e79e062c88
I20260620 08:14:09.097054 26735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791985de669f4265a2faf2dfce498f4c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 45401 } }
I20260620 08:14:09.097210 26735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:09.097254 26735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791985de669f4265a2faf2dfce498f4c, State: Initialized, Role: FOLLOWER
I20260620 08:14:09.097380 26735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791985de669f4265a2faf2dfce498f4c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 45401 } }
I20260620 08:14:09.097466 26735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:09.097510 26735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:09.097558 26735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:09.098260 26735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791985de669f4265a2faf2dfce498f4c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 45401 } }
I20260620 08:14:09.098367 26735 leader_election.cc:302] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 791985de669f4265a2faf2dfce498f4c; no voters:
I20260620 08:14:09.098515 26735 leader_election.cc:288] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:09.098567 26740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:09.098707 26740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [term 1 LEADER]: Becoming Leader. State: Replica: 791985de669f4265a2faf2dfce498f4c, State: Running, Role: LEADER
I20260620 08:14:09.098762 26735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:09.098819 26740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791985de669f4265a2faf2dfce498f4c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 45401 } }
I20260620 08:14:09.099159 26741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "791985de669f4265a2faf2dfce498f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791985de669f4265a2faf2dfce498f4c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 45401 } } }
I20260620 08:14:09.099287 26741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:09.099148 26740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 791985de669f4265a2faf2dfce498f4c. Latest consensus state: current_term: 1 leader_uuid: "791985de669f4265a2faf2dfce498f4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "791985de669f4265a2faf2dfce498f4c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 45401 } } }
I20260620 08:14:09.099469 26740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:09.099588 26749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:09.100091 26749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:09.101594 26749 catalog_manager.cc:1347] Generated new cluster ID: 0db60b2a33a94d50bfe68075219a608f
I20260620 08:14:09.101643 26749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:09.108580 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.096320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.24.249.190:45401)
0620 08:14:09.096525 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.096533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.096559 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:14:09.097012 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.097020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.097073 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.097213 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.098016 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.099474 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.100600 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.101225 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.101353 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.104238 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.104242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.104242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.104254 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.106872 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.107316 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.107319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.107319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.107434 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.107683 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.107684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.107684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.107977 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.108126 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.108246 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.108374 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260620 08:14:09.128340 26749 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:09.129101 26749 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:09.136621 26749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c: Generated new TSK 0
I20260620 08:14:09.136893 26749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:09.172732 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:45401
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:09.274514 26761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:09.274780 26761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:09.274823 26761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:09.276036 26761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:09.276104 26761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:09.276140 26761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:09.276175 26761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:09.277678 26761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:45401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:09.277997 26761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:09.278244 26761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:09.283782 26769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.283787 26766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.283819 26767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:09.284128 26761 server_base.cc:1034] running on GCE node
I20260620 08:14:09.284435 26761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:09.284691 26761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:09.285923 26761 hybrid_clock.cc:648] HybridClock initialized: now 1781943249285909 us; error 36 us; skew 500 ppm
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:09.288730 26761 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:09.289187 26761 webserver.cc:460] Webserver started at http://127.24.249.129:45689/ using document root <none> and password file <none>
I20260620 08:14:09.289458 26761 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:09.289527 26761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:09.289667 26761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:09.290779 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "0b89c546db554ceab398a5634669cb38"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "b10af8bf9ab9991b43abc4690eee9fb7"
server_key_iv: "6c36d861ef859347f10676bc5f0fdd1c"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.291133 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "0b89c546db554ceab398a5634669cb38"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "b10af8bf9ab9991b43abc4690eee9fb7"
server_key_iv: "6c36d861ef859347f10676bc5f0fdd1c"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.292600 26761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:09.293330 26776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.293509 26761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:09.293602 26761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "0b89c546db554ceab398a5634669cb38"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "b10af8bf9ab9991b43abc4690eee9fb7"
server_key_iv: "6c36d861ef859347f10676bc5f0fdd1c"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.293681 26761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:09.302666 26761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:09.304503 26761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:09.304677 26761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:09.305186 26761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:09.305246 26761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.305290 26761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:09.305330 26761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:09.312187 26761 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:44781
I20260620 08:14:09.312650 26761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:09.313089 26891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:44781 every 8 connection(s)
I20260620 08:14:09.315666 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.305744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:47459 (local address 127.24.249.190:45401)
0620 08:14:09.305803 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.305805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.306028 (+ 223us) server_negotiation.cc:407] Connection header received
0620 08:14:09.306555 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.306556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.306572 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.306635 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.307474 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.308376 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.309968 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.310246 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.310333 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.312620 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.312621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.312622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.312629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.314365 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.314801 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.314802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.314802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.314862 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.315078 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.315079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.315080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.315197 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.315256 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.315485 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.315559 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:09.315786 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.305717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:45401 (local address 127.24.249.129:47459)
0620 08:14:09.305906 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.305952 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.306449 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.306714 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.306722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.307084 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:09.307362 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.307367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.308475 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.308476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.309839 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.309843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.310343 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.310345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.310483 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.311040 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:09.311050 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:09.312507 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:09.314464 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.314471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.314472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.314727 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.314924 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.314925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.314925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.315020 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.315254 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:09.315258 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:09.315366 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.315449 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.315567 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260620 08:14:09.316365 26892 heartbeater.cc:347] Connected to a master server at 127.24.249.190:45401
I20260620 08:14:09.316447 26892 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:09.316630 26892 heartbeater.cc:511] Master 127.24.249.190:45401 requested a full tablet report, sending...
I20260620 08:14:09.317201 26688 ts_manager.cc:194] Registered new tserver with Master: 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781)
I20260620 08:14:09.318164 26688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:47459
I20260620 08:14:09.318593 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 26761
I20260620 08:14:09.318706 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:09.318876 25574 external_mini_cluster.cc:1428] Setting key 9b20d295b093b3316981ee4324c4b59d
I20260620 08:14:09.323824 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.317488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:39719 (local address 127.24.249.190:45401)
0620 08:14:09.317535 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.317537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.317620 (+ 83us) server_negotiation.cc:407] Connection header received
0620 08:14:09.317689 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.317690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.317698 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.317748 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.318118 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.318887 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.319734 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.319959 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.320043 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.321080 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.321082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.321082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.321090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.322620 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.323046 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.323047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.323047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.323097 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.323315 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.323317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.323317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.323433 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.323487 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.323697 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.323765 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:14:09.323931 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.317416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:45401 (local address 127.24.249.129:39719)
0620 08:14:09.317561 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.317573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.317634 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.317752 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.317753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.317972 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:09.318037 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.318039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.319012 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.319013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.319629 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.319632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.320035 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.320036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.320147 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.320659 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:09.320663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:09.320966 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:09.322701 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.322702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.322703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.322967 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.323160 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.323161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.323161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.323241 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.323502 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:09.323503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:09.323579 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.323677 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.323798 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:09.348726 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:45401
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:09.435721 26901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:09.435958 26901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:09.436000 26901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:09.437186 26901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:09.437253 26901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:09.437288 26901 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:09.437326 26901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:09.438858 26901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:45401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:09.439122 26901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:09.439357 26901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:09.444502 26906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.444496 26909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.444542 26907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:09.444839 26901 server_base.cc:1034] running on GCE node
I20260620 08:14:09.445086 26901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:09.445323 26901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:09.446516 26901 hybrid_clock.cc:648] HybridClock initialized: now 1781943249446485 us; error 33 us; skew 500 ppm
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:09.449492 26901 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:09.450058 26901 webserver.cc:460] Webserver started at http://127.24.249.130:39787/ using document root <none> and password file <none>
I20260620 08:14:09.450335 26901 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:09.450408 26901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:09.450567 26901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:09.451745 26901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "395e8593d8cd4563bae0a5e5c3e725c3"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "3175dbd1af88caa1e38217efbb39102a"
server_key_iv: "69082b7b86f1791e9919a41f87953ea8"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.452148 26901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "395e8593d8cd4563bae0a5e5c3e725c3"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "3175dbd1af88caa1e38217efbb39102a"
server_key_iv: "69082b7b86f1791e9919a41f87953ea8"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.453668 26901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:09.454536 26916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.454766 26901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:09.454859 26901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "395e8593d8cd4563bae0a5e5c3e725c3"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "3175dbd1af88caa1e38217efbb39102a"
server_key_iv: "69082b7b86f1791e9919a41f87953ea8"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.454937 26901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:09.468087 26901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:09.470568 26901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:09.470805 26901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:09.471402 26901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:09.471468 26901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.471515 26901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:09.471546 26901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.478266 26901 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:46243
I20260620 08:14:09.478336 27031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:46243 every 8 connection(s)
I20260620 08:14:09.478660 26901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:09.482487 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.471803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:53059 (local address 127.24.249.190:45401)
0620 08:14:09.471845 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.471847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.472083 (+ 236us) server_negotiation.cc:407] Connection header received
0620 08:14:09.472630 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.472632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.472645 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.472691 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.473548 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.474382 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.476152 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.476418 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.476478 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.479323 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.479325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.479325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.479332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.480812 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.481304 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.481305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.481306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.481362 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.481671 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.481672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.481672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.481957 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.482029 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.482345 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.482420 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:14:09.482628 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.471824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:45401 (local address 127.24.249.130:53059)
0620 08:14:09.471966 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.472001 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.472515 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.472765 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.472770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.473135 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:09.473423 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.473430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.474492 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.474493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.476031 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.476035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.476515 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.476516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.476687 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.477349 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:09.477362 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:09.479232 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:09.480899 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.480905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.480907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.481216 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.481446 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.481447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.481447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.481570 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.482032 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:09.482038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:09.482170 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.482287 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.482426 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260620 08:14:09.483218 27032 heartbeater.cc:347] Connected to a master server at 127.24.249.190:45401
I20260620 08:14:09.483327 27032 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:09.483530 27032 heartbeater.cc:511] Master 127.24.249.190:45401 requested a full tablet report, sending...
I20260620 08:14:09.484035 26688 ts_manager.cc:194] Registered new tserver with Master: 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130:46243)
I20260620 08:14:09.484184 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 26901
I20260620 08:14:09.484278 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:09.484472 25574 external_mini_cluster.cc:1428] Setting key 1b5ff1fb85a2e08bc9a83dc591133a00
I20260620 08:14:09.484613 26688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:53059
I20260620 08:14:09.492587 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.484811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:41925 (local address 127.24.249.190:45401)
0620 08:14:09.484855 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.484857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.485049 (+ 192us) server_negotiation.cc:407] Connection header received
0620 08:14:09.485101 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.485102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.485113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.485181 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.486400 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.487133 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.487877 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.488194 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.488285 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.489509 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.489511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.489511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.489519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.491229 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.491665 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.491666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.491666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.491725 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.492001 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.492002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.492002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.492165 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.492241 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.492445 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.492520 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:14:09.492789 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.484773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:45401 (local address 127.24.249.130:41925)
0620 08:14:09.484973 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.484984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.485046 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.485939 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.485940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.486165 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:09.486233 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.486235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.487244 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.487245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.487804 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.487805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.488298 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.488298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.488365 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.488917 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:09.488921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:09.489415 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:09.491320 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.491321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.491321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.491570 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.491810 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.491811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.491812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.491893 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.492253 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:09.492254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:09.492312 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.492395 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.492531 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:09.512974 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:45401
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:09.594771 27041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:09.594970 27041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:09.594993 27041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:09.596158 27041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:09.596199 27041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:09.596220 27041 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:09.596236 27041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:09.597746 27041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:45401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:09.598127 27041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:09.598322 27041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:09.603348 27047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.603340 27049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:09.603340 27046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:09.603546 27041 server_base.cc:1034] running on GCE node
I20260620 08:14:09.603852 27041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:09.604074 27041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:09.605237 27041 hybrid_clock.cc:648] HybridClock initialized: now 1781943249605206 us; error 41 us; skew 500 ppm
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:09.607736 27041 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:09.608212 27041 webserver.cc:460] Webserver started at http://127.24.249.131:43205/ using document root <none> and password file <none>
I20260620 08:14:09.608436 27041 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:09.608498 27041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:09.608630 27041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:09.609670 27041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "045d9e8640bd4df49ee686c86ebdd93c"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "cff6d9cc93bcf2b44814d6554cdd6425"
server_key_iv: "c7adf03608f7d399eef43d68c9143b59"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.610057 27041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "045d9e8640bd4df49ee686c86ebdd93c"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "cff6d9cc93bcf2b44814d6554cdd6425"
server_key_iv: "c7adf03608f7d399eef43d68c9143b59"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.611387 27041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:09.612120 27056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.612289 27041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:09.612381 27041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "045d9e8640bd4df49ee686c86ebdd93c"
format_stamp: "Formatted at 2026-06-20 08:14:09 on dist-test-slave-65w8"
server_key: "cff6d9cc93bcf2b44814d6554cdd6425"
server_key_iv: "c7adf03608f7d399eef43d68c9143b59"
server_key_version: "encryptionkey@0"
I20260620 08:14:09.612460 27041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:09.638015 27041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:09.639912 27041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:09.640066 27041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:09.640625 27041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:09.640686 27041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.640726 27041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:09.640751 27041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:09.647809 27041 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:46357
I20260620 08:14:09.648200 27041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:09.648198 27171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:46357 every 8 connection(s)
I20260620 08:14:09.650243 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.641007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:53839 (local address 127.24.249.190:45401)
0620 08:14:09.641055 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.641057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.641308 (+ 251us) server_negotiation.cc:407] Connection header received
0620 08:14:09.641778 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.641779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.641792 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.641910 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.642514 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.643337 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.644399 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.644625 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.644697 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.647342 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.647343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.647344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.647351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.649027 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.649386 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.649387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.649387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.649438 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.649640 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.649641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.649641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.649780 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.649832 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.650112 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.650182 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:09.650403 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.641001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:45401 (local address 127.24.249.131:53839)
0620 08:14:09.641175 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.641216 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.641689 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.641971 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.641975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.642209 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:09.642406 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.642411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.643422 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.643424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.644296 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.644299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.644715 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.644716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.644883 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.645410 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:09.645417 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:09.647246 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:09.649096 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.649100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.649101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.649316 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.649494 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.649495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.649495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.649587 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.649835 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:09.649839 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:09.649994 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.650082 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.650198 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260620 08:14:09.650866 27172 heartbeater.cc:347] Connected to a master server at 127.24.249.190:45401
I20260620 08:14:09.650946 27172 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:09.651134 27172 heartbeater.cc:511] Master 127.24.249.190:45401 requested a full tablet report, sending...
I20260620 08:14:09.651530 26689 ts_manager.cc:194] Registered new tserver with Master: 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357)
I20260620 08:14:09.652015 26689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:53839
I20260620 08:14:09.657073 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.651941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:47305 (local address 127.24.249.190:45401)
0620 08:14:09.651981 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.651983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.652125 (+ 142us) server_negotiation.cc:407] Connection header received
0620 08:14:09.652299 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.652300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.652308 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.652361 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.652730 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.653193 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.653922 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.654122 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.654180 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.655026 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.655027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.655028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.655033 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.656237 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.656521 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.656522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.656522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.656574 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.656746 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.656746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.656746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.656834 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.656878 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.656964 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.657017 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:09.657084 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.651889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:45401 (local address 127.24.249.131:47305)
0620 08:14:09.652038 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.652048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.652120 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.652379 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.652380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.652575 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:09.652635 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.652636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.653298 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.653299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.653787 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.653789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.654196 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.654197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.654269 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.654677 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:09.654681 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:09.654949 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:09.656307 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.656308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.656308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.656460 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.656658 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:09.656659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:09.656659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:09.656707 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:09.656872 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:09.656872 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:09.656903 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.656949 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.657018 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260620 08:14:09.658445 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 27041
I20260620 08:14:09.658582 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:09.658799 25574 external_mini_cluster.cc:1428] Setting key e5dcf3e6b996d89e623efc7f66f74e0f
I20260620 08:14:09.659484 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:14:09 dist-test-slave-65w8 krb5kdc[26644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:09.683228 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.675481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.24.249.190:45401)
0620 08:14:09.675558 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.675560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.675569 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:09.675603 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.675604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.675614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.675683 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:09.676314 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.676859 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.677678 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.677975 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.678062 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.680479 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:09.680481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:09.680481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:09.680488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:09.682003 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.682369 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.682370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.682370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.682426 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:09.682693 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:09.682695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:09.682695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:09.682821 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:09.682913 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.683025 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.683093 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:14:09.685154 26689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:09.687139 26689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:09.694306 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.690247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46243 (local address 127.0.0.1:55310)
0620 08:14:09.690348 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.690359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.690412 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.690759 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.690761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.690771 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:09.690838 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.690840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.692118 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.692119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.693138 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.693141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.693962 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.693963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.694046 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.694063 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.694152 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.694206 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":49,"thread_start_us":61,"threads_started":1}
I20260620 08:14:09.694324 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.690404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.24.249.130:46243)
0620 08:14:09.690577 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.690589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.690599 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:09.690635 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.690637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.690651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.690755 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:09.690916 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.691993 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.693223 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.693817 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.693938 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.694001 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.694173 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.694215 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260620 08:14:09.694823 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.690151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.24.249.129:44781)
0620 08:14:09.690672 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.690677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.690689 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:09.690734 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.690736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.690753 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.690838 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:09.691045 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.691851 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.693584 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.694412 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.694496 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.694546 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.694677 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.694719 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":77,"threads_started":1}
I20260620 08:14:09.694911 27191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.690039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:44781 (local address 127.0.0.1:35466)
0620 08:14:09.690186 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.690196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.690245 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.690926 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.690927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.690928 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:09.690959 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.690960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.691944 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.691946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.693457 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.693459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.694510 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.694511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.694590 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.694596 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.694649 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.694688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":75,"thread_start_us":108,"threads_started":1}
I20260620 08:14:09.695292 26963 tablet_service.cc:1467] Processing CreateTablet for tablet 680c3ff063e14eadbbee6f7cad8afc31 (DEFAULT_TABLE table=test-table [id=8411955abc28453da5bbed11a2c8dd07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:09.695667 26963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680c3ff063e14eadbbee6f7cad8afc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:09.695569 26826 tablet_service.cc:1467] Processing CreateTablet for tablet 680c3ff063e14eadbbee6f7cad8afc31 (DEFAULT_TABLE table=test-table [id=8411955abc28453da5bbed11a2c8dd07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:09.695901 26826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680c3ff063e14eadbbee6f7cad8afc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:09.700052 27195 tablet_bootstrap.cc:492] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Bootstrap starting.
I20260620 08:14:09.700054 27196 tablet_bootstrap.cc:492] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Bootstrap starting.
I20260620 08:14:09.700750 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.689801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46357 (local address 127.0.0.1:47852)
0620 08:14:09.689950 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.690010 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.690096 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.690632 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.690635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.690638 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:09.690718 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.690721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.692019 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.692019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.693577 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.693579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.700422 (+ 6843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.700423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.700510 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.700515 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.700611 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.700667 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":42,"thread_start_us":74,"threads_started":1}
I20260620 08:14:09.700819 27196 tablet_bootstrap.cc:654] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:09.700819 27195 tablet_bootstrap.cc:654] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:09.700775 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.689924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.24.249.131:46357)
0620 08:14:09.690397 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.690403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.690414 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:09.690455 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.690457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.690473 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.690548 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:09.690799 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.691944 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.693655 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.700330 (+ 6675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.700424 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.700486 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.700630 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.700679 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":109,"threads_started":1}
I20260620 08:14:09.701270 27196 log.cc:824] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:09.701479 27195 log.cc:824] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:09.701685 27093 tablet_service.cc:1467] Processing CreateTablet for tablet 680c3ff063e14eadbbee6f7cad8afc31 (DEFAULT_TABLE table=test-table [id=8411955abc28453da5bbed11a2c8dd07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:09.702097 27093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680c3ff063e14eadbbee6f7cad8afc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:09.702147 27196 tablet_bootstrap.cc:492] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: No bootstrap required, opened a new log
I20260620 08:14:09.702210 27196 ts_tablet_manager.cc:1397] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:09.702419 27195 tablet_bootstrap.cc:492] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: No bootstrap required, opened a new log
I20260620 08:14:09.702549 27195 ts_tablet_manager.cc:1397] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:14:09.703581 27196 raft_consensus.cc:348] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.703717 27196 raft_consensus.cc:374] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:09.703739 27196 raft_consensus.cc:729] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395e8593d8cd4563bae0a5e5c3e725c3, State: Initialized, Role: FOLLOWER
I20260620 08:14:09.703866 27196 consensus_queue.cc:260] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.704144 27196 ts_tablet_manager.cc:1428] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:14:09.704154 27032 heartbeater.cc:503] Master 127.24.249.190:45401 was elected leader, sending a full tablet report...
I20260620 08:14:09.705098 27195 raft_consensus.cc:348] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.705271 27195 raft_consensus.cc:374] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:09.705304 27195 raft_consensus.cc:729] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b89c546db554ceab398a5634669cb38, State: Initialized, Role: FOLLOWER
I20260620 08:14:09.705421 27195 consensus_queue.cc:260] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.705765 27200 tablet_bootstrap.cc:492] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Bootstrap starting.
I20260620 08:14:09.705854 27195 ts_tablet_manager.cc:1428] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:14:09.705920 26892 heartbeater.cc:503] Master 127.24.249.190:45401 was elected leader, sending a full tablet report...
I20260620 08:14:09.706868 27200 tablet_bootstrap.cc:654] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:09.707177 27200 log.cc:824] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:09.707929 27200 tablet_bootstrap.cc:492] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: No bootstrap required, opened a new log
I20260620 08:14:09.708011 27200 ts_tablet_manager.cc:1397] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:09.709394 27200 raft_consensus.cc:348] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.709542 27200 raft_consensus.cc:374] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:09.709586 27200 raft_consensus.cc:729] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045d9e8640bd4df49ee686c86ebdd93c, State: Initialized, Role: FOLLOWER
I20260620 08:14:09.709720 27200 consensus_queue.cc:260] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.710011 27172 heartbeater.cc:503] Master 127.24.249.190:45401 was elected leader, sending a full tablet report...
I20260620 08:14:09.709995 27200 ts_tablet_manager.cc:1428] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260620 08:14:09.729339 27033 tablet.cc:2369] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:09.751797 27201 raft_consensus.cc:482] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:09.751926 27201 raft_consensus.cc:504] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.752219 27201 leader_election.cc:288] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357), 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130:46243)
I20260620 08:14:09.756095 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.752497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:43417 (local address 127.24.249.131:46357)
0620 08:14:09.752547 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.752549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.752558 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:09.752595 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.752596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.752647 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.752712 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:09.752914 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.753666 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.755153 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.755768 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.755833 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.755840 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.755987 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.756026 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 08:14:09.756095 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.752610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:49751 (local address 127.24.249.130:46243)
0620 08:14:09.752674 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.752676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.752763 (+ 87us) server_negotiation.cc:407] Connection header received
0620 08:14:09.752814 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.752815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.752822 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.752862 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:09.753006 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.753708 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.755153 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.755709 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.755798 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.755809 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.755987 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.756036 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:14:09.756104 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.752407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46357 (local address 127.24.249.129:43417)
0620 08:14:09.752464 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.752477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.752532 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.752738 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.752739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.752741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:09.752818 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.752820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.753765 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.753766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.755034 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.755037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.755837 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.755837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.755917 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.755920 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.755972 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.756007 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":9}
I20260620 08:14:09.756117 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.752505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46243 (local address 127.24.249.129:49751)
0620 08:14:09.752678 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:09.752701 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:09.752759 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:09.752880 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:09.752881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:09.752883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:09.752942 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.752944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.753775 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.753776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.755034 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:09.755037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.755819 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.755820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:09.755897 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.755902 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:09.755966 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:14:09.756007 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":52,"thread_start_us":106,"threads_started":1}
I20260620 08:14:09.756508 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c3ff063e14eadbbee6f7cad8afc31" candidate_uuid: "0b89c546db554ceab398a5634669cb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" is_pre_election: true
I20260620 08:14:09.756508 27126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c3ff063e14eadbbee6f7cad8afc31" candidate_uuid: "0b89c546db554ceab398a5634669cb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045d9e8640bd4df49ee686c86ebdd93c" is_pre_election: true
I20260620 08:14:09.756660 27126 raft_consensus.cc:2454] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b89c546db554ceab398a5634669cb38 in term 0.
I20260620 08:14:09.756660 26986 raft_consensus.cc:2454] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b89c546db554ceab398a5634669cb38 in term 0.
I20260620 08:14:09.756933 26780 leader_election.cc:302] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045d9e8640bd4df49ee686c86ebdd93c, 0b89c546db554ceab398a5634669cb38; no voters:
I20260620 08:14:09.757079 27201 raft_consensus.cc:2781] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:09.757134 27201 raft_consensus.cc:482] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:09.757159 27201 raft_consensus.cc:3037] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:09.758256 27201 raft_consensus.cc:504] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.758384 27201 leader_election.cc:288] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [CANDIDATE]: Term 1 election: Requested vote from peers 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357), 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130:46243)
I20260620 08:14:09.758581 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c3ff063e14eadbbee6f7cad8afc31" candidate_uuid: "0b89c546db554ceab398a5634669cb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395e8593d8cd4563bae0a5e5c3e725c3"
I20260620 08:14:09.758599 27126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c3ff063e14eadbbee6f7cad8afc31" candidate_uuid: "0b89c546db554ceab398a5634669cb38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045d9e8640bd4df49ee686c86ebdd93c"
I20260620 08:14:09.758644 26986 raft_consensus.cc:3037] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:09.758702 27126 raft_consensus.cc:3037] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:09.759316 26986 raft_consensus.cc:2454] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b89c546db554ceab398a5634669cb38 in term 1.
I20260620 08:14:09.759472 26780 leader_election.cc:302] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b89c546db554ceab398a5634669cb38, 395e8593d8cd4563bae0a5e5c3e725c3; no voters:
I20260620 08:14:09.759534 27126 raft_consensus.cc:2454] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b89c546db554ceab398a5634669cb38 in term 1.
I20260620 08:14:09.759572 27201 raft_consensus.cc:2781] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:09.759719 27201 raft_consensus.cc:686] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 1 LEADER]: Becoming Leader. State: Replica: 0b89c546db554ceab398a5634669cb38, State: Running, Role: LEADER
I20260620 08:14:09.759820 27201 consensus_queue.cc:237] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:09.760686 26688 catalog_manager.cc:5495] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b89c546db554ceab398a5634669cb38 (127.24.249.129). New cstate: current_term: 1 leader_uuid: "0b89c546db554ceab398a5634669cb38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:09.768581 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:09.766127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.24.249.129:44781)
0620 08:14:09.766203 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:09.766204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:09.766227 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:14:09.766309 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:09.766310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:09.766324 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:09.766382 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:09.766532 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.766976 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.767738 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:09.767937 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:09.767998 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:09.768224 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:09.768320 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:09.768390 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:09.768452 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 08:14:09.768470 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:14:09.770301 26986 raft_consensus.cc:1261] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Refusing update from remote peer 0b89c546db554ceab398a5634669cb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:09.770339 27126 raft_consensus.cc:1261] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Refusing update from remote peer 0b89c546db554ceab398a5634669cb38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:09.770696 27201 consensus_queue.cc:1035] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:09.770819 27205 consensus_queue.cc:1035] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:09.773566 27209 mvcc.cc:205] Tried to move back new op lower bound from 7298839551056408576 to 7298839551016869888. Current Snapshot: MvccSnapshot[applied={T|T < 7298839551056408576}]
I20260620 08:14:09.777006 26688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:09.777098 26688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:09.777971 26688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c: Sending DeleteTablet for 3 replicas of tablet 680c3ff063e14eadbbee6f7cad8afc31
I20260620 08:14:09.778334 27093 tablet_service.cc:1514] Processing DeleteTablet for tablet 680c3ff063e14eadbbee6f7cad8afc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:09 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:47852
I20260620 08:14:09.778349 26963 tablet_service.cc:1514] Processing DeleteTablet for tablet 680c3ff063e14eadbbee6f7cad8afc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:09 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:55310
I20260620 08:14:09.778563 26826 tablet_service.cc:1514] Processing DeleteTablet for tablet 680c3ff063e14eadbbee6f7cad8afc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:09 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:35466
I20260620 08:14:09.778707 26688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:14:09.778784 27220 tablet_replica.cc:330] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: stopping tablet replica
W20260620 08:14:09.778867 26688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:09.778883 27220 raft_consensus.cc:2229] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:09.778999 27220 raft_consensus.cc:2258] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:09.779700 27220 ts_tablet_manager.cc:1905] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:09.780099 26963 tablet_service.cc:1467] Processing CreateTablet for tablet dcf08c25a82047d4b40761b6800247f3 (DEFAULT_TABLE table=test-table [id=fecec18b6f9e4bdaa5408b5c04ea8044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:09.780254 26963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf08c25a82047d4b40761b6800247f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:09.781490 27196 tablet_bootstrap.cc:492] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: Bootstrap starting.
I20260620 08:14:09.781612 26826 tablet_service.cc:1467] Processing CreateTablet for tablet dcf08c25a82047d4b40761b6800247f3 (DEFAULT_TABLE table=test-table [id=fecec18b6f9e4bdaa5408b5c04ea8044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:09.781730 26826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf08c25a82047d4b40761b6800247f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:09.781937 27196 tablet_bootstrap.cc:654] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:09.782078 27220 ts_tablet_manager.cc:1918] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:09.782140 27220 log.cc:1196] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/680c3ff063e14eadbbee6f7cad8afc31
I20260620 08:14:09.782401 27220 ts_tablet_manager.cc:1939] T 680c3ff063e14eadbbee6f7cad8afc31 P 0b89c546db554ceab398a5634669cb38: Deleting consensus metadata
I20260620 08:14:09.783020 27195 tablet_bootstrap.cc:492] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Bootstrap starting.
I20260620 08:14:09.783092 27093 tablet_service.cc:1467] Processing CreateTablet for tablet dcf08c25a82047d4b40761b6800247f3 (DEFAULT_TABLE table=test-table [id=fecec18b6f9e4bdaa5408b5c04ea8044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:09.783223 27093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf08c25a82047d4b40761b6800247f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:09.783442 27195 tablet_bootstrap.cc:654] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:09.784142 27195 tablet_bootstrap.cc:492] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: No bootstrap required, opened a new log
I20260620 08:14:09.784201 27195 ts_tablet_manager.cc:1397] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:09.784375 27195 raft_consensus.cc:348] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:09.784469 27195 raft_consensus.cc:374] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:09.784495 27195 raft_consensus.cc:729] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b89c546db554ceab398a5634669cb38, State: Initialized, Role: FOLLOWER
I20260620 08:14:09.784544 27195 consensus_queue.cc:260] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:09.784628 27195 ts_tablet_manager.cc:1428] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.784811 27219 tablet_replica.cc:330] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: stopping tablet replica
I20260620 08:14:09.784936 27219 raft_consensus.cc:2229] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:09.784965 27200 tablet_bootstrap.cc:492] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Bootstrap starting.
I20260620 08:14:09.785145 26676 catalog_manager.cc:4841] TS 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781): tablet 680c3ff063e14eadbbee6f7cad8afc31 (table test-table [id=8411955abc28453da5bbed11a2c8dd07]) successfully deleted
I20260620 08:14:09.785753 27196 tablet_bootstrap.cc:492] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: No bootstrap required, opened a new log
I20260620 08:14:09.785804 27196 ts_tablet_manager.cc:1397] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 08:14:09.786023 27196 raft_consensus.cc:348] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:09.786091 27196 raft_consensus.cc:374] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:09.786105 27196 raft_consensus.cc:729] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395e8593d8cd4563bae0a5e5c3e725c3, State: Initialized, Role: FOLLOWER
I20260620 08:14:09.786161 27196 consensus_queue.cc:260] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:09.786255 27196 ts_tablet_manager.cc:1428] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.785049 27219 raft_consensus.cc:2258] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:09.785359 27200 tablet_bootstrap.cc:654] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:09.793080 27218 tablet_replica.cc:330] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: stopping tablet replica
I20260620 08:14:09.793202 27219 ts_tablet_manager.cc:1905] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:09.793200 27218 raft_consensus.cc:2229] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:09.793440 27218 raft_consensus.cc:2258] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:09.793454 27200 tablet_bootstrap.cc:492] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: No bootstrap required, opened a new log
I20260620 08:14:09.793497 27200 ts_tablet_manager.cc:1397] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260620 08:14:09.793630 27200 raft_consensus.cc:348] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:09.793696 27200 raft_consensus.cc:374] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:09.793716 27200 raft_consensus.cc:729] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045d9e8640bd4df49ee686c86ebdd93c, State: Initialized, Role: FOLLOWER
I20260620 08:14:09.793769 27200 consensus_queue.cc:260] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:09.793843 27200 ts_tablet_manager.cc:1428] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:09.793915 27218 ts_tablet_manager.cc:1905] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:09.795168 27219 ts_tablet_manager.cc:1918] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:09.795257 27219 log.cc:1196] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/680c3ff063e14eadbbee6f7cad8afc31
I20260620 08:14:09.795524 27219 ts_tablet_manager.cc:1939] T 680c3ff063e14eadbbee6f7cad8afc31 P 045d9e8640bd4df49ee686c86ebdd93c: Deleting consensus metadata
I20260620 08:14:09.795830 27218 ts_tablet_manager.cc:1918] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:09.795897 27218 log.cc:1196] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/680c3ff063e14eadbbee6f7cad8afc31
I20260620 08:14:09.795944 26676 catalog_manager.cc:4841] TS 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357): tablet 680c3ff063e14eadbbee6f7cad8afc31 (table test-table [id=8411955abc28453da5bbed11a2c8dd07]) successfully deleted
I20260620 08:14:09.796169 27218 ts_tablet_manager.cc:1939] T 680c3ff063e14eadbbee6f7cad8afc31 P 395e8593d8cd4563bae0a5e5c3e725c3: Deleting consensus metadata
I20260620 08:14:09.796552 26676 catalog_manager.cc:4841] TS 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130:46243): tablet 680c3ff063e14eadbbee6f7cad8afc31 (table test-table [id=8411955abc28453da5bbed11a2c8dd07]) successfully deleted
W20260620 08:14:09.813973 26893 tablet.cc:2369] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:14:09.898940 27173 tablet.cc:2369] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:10.066519 27203 raft_consensus.cc:482] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:10.066644 27203 raft_consensus.cc:504] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:10.067039 27203 leader_election.cc:288] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781), 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130:46243)
I20260620 08:14:10.067770 27199 raft_consensus.cc:482] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:10.067843 27199 raft_consensus.cc:504] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:10.068133 27199 leader_election.cc:288] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781), 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357)
I20260620 08:14:10.070887 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.067229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:44781 (local address 127.24.249.131:52365)
0620 08:14:10.067293 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.067306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.067376 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.067541 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.067542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.067544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:10.067625 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.067627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.068493 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.068495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.069793 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.069796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.070598 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.070599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.070715 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.070718 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.070766 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.070810 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":9}
I20260620 08:14:10.071166 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.067296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:52365 (local address 127.24.249.129:44781)
0620 08:14:10.067343 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.067344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.067358 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:14:10.067461 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.067462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.067475 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.067530 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:10.067709 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.068390 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.069942 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.070491 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.070560 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.070569 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.071046 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.071100 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:14:10.071306 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.067455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:40151 (local address 127.24.249.130:46243)
0620 08:14:10.067496 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.067498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.067574 (+ 76us) server_negotiation.cc:407] Connection header received
0620 08:14:10.067624 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.067625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.067632 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.067703 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:10.067929 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.068715 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.070268 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.070855 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.070908 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.070916 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.071180 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.071223 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:10.071345 26846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "045d9e8640bd4df49ee686c86ebdd93c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b89c546db554ceab398a5634669cb38" is_pre_election: true
I20260620 08:14:10.071477 26846 raft_consensus.cc:2454] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045d9e8640bd4df49ee686c86ebdd93c in term 0.
I20260620 08:14:10.071600 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.067387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46243 (local address 127.24.249.131:40151)
0620 08:14:10.067492 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.067504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.067568 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.067735 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.067736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.067738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:10.067814 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.067815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.068827 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.068828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.070163 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.070165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.071022 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.071024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.071095 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.071098 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.071441 (+ 343us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.071492 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260620 08:14:10.071666 27060 leader_election.cc:302] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045d9e8640bd4df49ee686c86ebdd93c, 0b89c546db554ceab398a5634669cb38; no voters:
I20260620 08:14:10.071871 27203 raft_consensus.cc:2781] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:10.071915 27203 raft_consensus.cc:482] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:10.071936 27203 raft_consensus.cc:3037] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:10.071991 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.068275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:44781 (local address 127.24.249.130:38467)
0620 08:14:10.068427 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.068436 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.068485 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.068803 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.068804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.068806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:10.068867 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.068868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.069788 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.069789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.071017 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.071021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.071755 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.071756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.071833 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.071837 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.071868 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.071910 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260620 08:14:10.072240 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.068319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:38467 (local address 127.24.249.129:44781)
0620 08:14:10.068683 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.068685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.068691 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:14:10.068710 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.068711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.068720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.068928 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:10.068960 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.069695 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.071127 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.071663 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.072097 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.072105 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.072125 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.072165 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":58,"threads_started":1}
I20260620 08:14:10.072423 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "045d9e8640bd4df49ee686c86ebdd93c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" is_pre_election: true
I20260620 08:14:10.072507 26986 raft_consensus.cc:2454] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045d9e8640bd4df49ee686c86ebdd93c in term 0.
I20260620 08:14:10.072613 26846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b89c546db554ceab398a5634669cb38" is_pre_election: true
I20260620 08:14:10.072693 26846 raft_consensus.cc:2454] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395e8593d8cd4563bae0a5e5c3e725c3 in term 0.
I20260620 08:14:10.072871 26920 leader_election.cc:302] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b89c546db554ceab398a5634669cb38, 395e8593d8cd4563bae0a5e5c3e725c3; no voters:
I20260620 08:14:10.072997 27199 raft_consensus.cc:2781] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:10.072965 27203 raft_consensus.cc:504] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:10.073029 27199 raft_consensus.cc:482] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:10.073055 27199 raft_consensus.cc:3037] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:10.073120 27203 leader_election.cc:288] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [CANDIDATE]: Term 1 election: Requested vote from peers 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781), 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130:46243)
I20260620 08:14:10.073282 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.068580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:36313 (local address 127.24.249.131:46357)
0620 08:14:10.068618 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.068620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.068628 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:10.068650 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.068651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.068658 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.068709 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:10.068943 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.069731 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.072075 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.072755 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.072803 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.072811 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.073133 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.073175 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:10.073300 26846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "045d9e8640bd4df49ee686c86ebdd93c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b89c546db554ceab398a5634669cb38"
I20260620 08:14:10.073323 26986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "045d9e8640bd4df49ee686c86ebdd93c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395e8593d8cd4563bae0a5e5c3e725c3"
I20260620 08:14:10.073369 26846 raft_consensus.cc:3037] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:10.073553 27227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.068441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46357 (local address 127.24.249.130:36313)
0620 08:14:10.068566 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.068574 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.068607 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.068782 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.068783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.068785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:10.068856 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.068858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.069845 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.069846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.071962 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.071965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.072936 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.072937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.073024 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.073028 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.073390 (+ 362us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.073441 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":36,"threads_started":1}
I20260620 08:14:10.073808 27126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045d9e8640bd4df49ee686c86ebdd93c" is_pre_election: true
I20260620 08:14:10.073922 27126 raft_consensus.cc:2379] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 395e8593d8cd4563bae0a5e5c3e725c3 in current term 1: Already voted for candidate 045d9e8640bd4df49ee686c86ebdd93c in this term.
I20260620 08:14:10.073966 27199 raft_consensus.cc:504] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:10.074076 27199 leader_election.cc:288] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781), 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357)
I20260620 08:14:10.074064 26986 raft_consensus.cc:2379] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 045d9e8640bd4df49ee686c86ebdd93c in current term 1: Already voted for candidate 395e8593d8cd4563bae0a5e5c3e725c3 in this term.
I20260620 08:14:10.074240 26845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b89c546db554ceab398a5634669cb38"
I20260620 08:14:10.074404 26846 raft_consensus.cc:2454] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045d9e8640bd4df49ee686c86ebdd93c in term 1.
I20260620 08:14:10.074363 27126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf08c25a82047d4b40761b6800247f3" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045d9e8640bd4df49ee686c86ebdd93c"
I20260620 08:14:10.074455 27126 raft_consensus.cc:2379] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 395e8593d8cd4563bae0a5e5c3e725c3 in current term 1: Already voted for candidate 045d9e8640bd4df49ee686c86ebdd93c in this term.
I20260620 08:14:10.074538 26920 leader_election.cc:302] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 395e8593d8cd4563bae0a5e5c3e725c3; no voters: 045d9e8640bd4df49ee686c86ebdd93c, 0b89c546db554ceab398a5634669cb38
I20260620 08:14:10.074559 27060 leader_election.cc:302] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045d9e8640bd4df49ee686c86ebdd93c, 0b89c546db554ceab398a5634669cb38; no voters:
I20260620 08:14:10.074630 27199 raft_consensus.cc:2726] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 08:14:10.074647 27203 raft_consensus.cc:2781] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:10.074848 27203 raft_consensus.cc:686] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 LEADER]: Becoming Leader. State: Replica: 045d9e8640bd4df49ee686c86ebdd93c, State: Running, Role: LEADER
I20260620 08:14:10.074909 27203 consensus_queue.cc:237] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } }
I20260620 08:14:10.075591 26688 catalog_manager.cc:5495] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c reported cstate change: term changed from 0 to 1, leader changed from <none> to 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131). New cstate: current_term: 1 leader_uuid: "045d9e8640bd4df49ee686c86ebdd93c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } health_report { overall_health: HEALTHY } } }
Jun 20 08:14:10 dist-test-slave-65w8 krb5kdc[26644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:10.161979 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.154603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:45401 (local address 127.0.0.1:33456)
0620 08:14:10.154802 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.154815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.154906 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.155053 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.155054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.155399 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:10.155446 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.155448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.156138 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.156139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.156674 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.156676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.157085 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.157086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.157142 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.157642 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:10.157646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:10.159237 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:10.160903 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:10.160910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:10.160913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:10.161114 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:10.161297 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:10.161298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:10.161298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:10.161356 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:10.161556 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:10.161563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:10.161605 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.161719 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.161815 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260620 08:14:10.161988 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.154663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.24.249.190:45401)
0620 08:14:10.154786 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.154788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.154885 (+ 97us) server_negotiation.cc:407] Connection header received
0620 08:14:10.154975 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.154975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.154986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.155033 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:10.155543 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.156057 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.156769 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.157006 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.157084 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.159333 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:10.159334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:10.159334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:10.159341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:10.160792 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:10.161190 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:10.161190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:10.161191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:10.161242 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:10.161408 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:10.161408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:10.161408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:10.161508 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:10.161556 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.161808 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.161917 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 08:14:10.163910 26688 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:33456:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:14:10.164095 26688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:10.165462 27093 tablet_service.cc:1467] Processing CreateTablet for tablet 29106fecffeb4bf3bcc8f225351e7b23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65ddbd2361184e78a649c2c6dbd1a7cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:10.165498 26963 tablet_service.cc:1467] Processing CreateTablet for tablet 29106fecffeb4bf3bcc8f225351e7b23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65ddbd2361184e78a649c2c6dbd1a7cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:10.165619 27093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29106fecffeb4bf3bcc8f225351e7b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:10.165648 26963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29106fecffeb4bf3bcc8f225351e7b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:10.166800 27200 tablet_bootstrap.cc:492] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: Bootstrap starting.
I20260620 08:14:10.166914 26826 tablet_service.cc:1467] Processing CreateTablet for tablet 29106fecffeb4bf3bcc8f225351e7b23 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65ddbd2361184e78a649c2c6dbd1a7cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:10.167064 26826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29106fecffeb4bf3bcc8f225351e7b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:10.167191 27200 tablet_bootstrap.cc:654] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:10.167834 27200 tablet_bootstrap.cc:492] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: No bootstrap required, opened a new log
I20260620 08:14:10.167891 27200 ts_tablet_manager.cc:1397] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:10.168109 27200 raft_consensus.cc:348] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.168191 27200 raft_consensus.cc:374] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:10.168212 27200 raft_consensus.cc:729] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045d9e8640bd4df49ee686c86ebdd93c, State: Initialized, Role: FOLLOWER
I20260620 08:14:10.168255 27200 consensus_queue.cc:260] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.168373 27200 ts_tablet_manager.cc:1428] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:10.168424 27203 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.168538 27203 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:10.168639 27195 tablet_bootstrap.cc:492] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: Bootstrap starting.
I20260620 08:14:10.168680 27196 tablet_bootstrap.cc:492] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: Bootstrap starting.
I20260620 08:14:10.168991 27195 tablet_bootstrap.cc:654] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:10.169059 27196 tablet_bootstrap.cc:654] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:10.169510 27195 tablet_bootstrap.cc:492] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: No bootstrap required, opened a new log
I20260620 08:14:10.169564 27195 ts_tablet_manager.cc:1397] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:10.169682 27195 raft_consensus.cc:348] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.169765 27195 raft_consensus.cc:374] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:10.169781 27195 raft_consensus.cc:729] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b89c546db554ceab398a5634669cb38, State: Initialized, Role: FOLLOWER
I20260620 08:14:10.169840 27195 consensus_queue.cc:260] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.169958 27195 ts_tablet_manager.cc:1428] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:10.169996 27215 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.170105 27215 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:10.170327 27196 tablet_bootstrap.cc:492] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: No bootstrap required, opened a new log
I20260620 08:14:10.170383 27196 ts_tablet_manager.cc:1397] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:10.170532 27196 raft_consensus.cc:348] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.170601 27196 raft_consensus.cc:374] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:10.170625 27196 raft_consensus.cc:729] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395e8593d8cd4563bae0a5e5c3e725c3, State: Initialized, Role: FOLLOWER
I20260620 08:14:10.170688 27196 consensus_queue.cc:260] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.170782 27196 ts_tablet_manager.cc:1428] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:10.170861 27199 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.170984 27199 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:10.339181 27199 raft_consensus.cc:482] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:10.339293 27199 raft_consensus.cc:504] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.339440 27199 leader_election.cc:288] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781), 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357)
I20260620 08:14:10.339722 26846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29106fecffeb4bf3bcc8f225351e7b23" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b89c546db554ceab398a5634669cb38" is_pre_election: true
I20260620 08:14:10.339843 26846 raft_consensus.cc:2454] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395e8593d8cd4563bae0a5e5c3e725c3 in term 0.
I20260620 08:14:10.339803 27126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29106fecffeb4bf3bcc8f225351e7b23" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045d9e8640bd4df49ee686c86ebdd93c" is_pre_election: true
I20260620 08:14:10.339890 27126 raft_consensus.cc:2454] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 395e8593d8cd4563bae0a5e5c3e725c3 in term 0.
I20260620 08:14:10.340018 26920 leader_election.cc:302] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b89c546db554ceab398a5634669cb38, 395e8593d8cd4563bae0a5e5c3e725c3; no voters:
I20260620 08:14:10.340147 27199 raft_consensus.cc:2781] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:10.340197 27199 raft_consensus.cc:482] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:10.340220 27199 raft_consensus.cc:3037] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:10.340870 27199 raft_consensus.cc:504] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.340999 27199 leader_election.cc:288] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781), 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357)
I20260620 08:14:10.341194 26846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29106fecffeb4bf3bcc8f225351e7b23" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b89c546db554ceab398a5634669cb38"
I20260620 08:14:10.341266 26846 raft_consensus.cc:3037] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:10.341215 27126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29106fecffeb4bf3bcc8f225351e7b23" candidate_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045d9e8640bd4df49ee686c86ebdd93c"
I20260620 08:14:10.341333 27126 raft_consensus.cc:3037] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:10.341936 26846 raft_consensus.cc:2454] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395e8593d8cd4563bae0a5e5c3e725c3 in term 1.
I20260620 08:14:10.342083 27126 raft_consensus.cc:2454] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 395e8593d8cd4563bae0a5e5c3e725c3 in term 1.
I20260620 08:14:10.342109 26920 leader_election.cc:302] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b89c546db554ceab398a5634669cb38, 395e8593d8cd4563bae0a5e5c3e725c3; no voters:
I20260620 08:14:10.342183 27199 raft_consensus.cc:2781] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:10.342316 27199 raft_consensus.cc:686] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 LEADER]: Becoming Leader. State: Replica: 395e8593d8cd4563bae0a5e5c3e725c3, State: Running, Role: LEADER
I20260620 08:14:10.342399 27199 consensus_queue.cc:237] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } }
I20260620 08:14:10.342732 27239 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: TxnStatusTablet state changed. Reason: New leader 395e8593d8cd4563bae0a5e5c3e725c3. Latest consensus state: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.342816 27239 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:10.342996 27241 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:14:10.343137 27241 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:14:10.343144 26689 catalog_manager.cc:5495] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130). New cstate: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } health_report { overall_health: HEALTHY } } }
I20260620 08:14:10.354209 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.351838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.24.249.130:46243)
0620 08:14:10.351893 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.351895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.351902 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:10.351934 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.351935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.351950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.352021 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:10.352151 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.352622 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.353365 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.353552 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.353650 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.353824 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:10.353978 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:10.354055 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.354118 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.354137 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:10.354247 27235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.351611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46243 (local address 127.0.0.1:55316)
0620 08:14:10.351682 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.351695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.351751 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.352028 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.352030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.352031 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:10.352081 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.352083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.352710 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.352711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.353285 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.353287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.353650 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.353651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.353713 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.353726 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:10.354040 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:10.354050 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.354107 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.354156 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 08:14:10.355427 27126 raft_consensus.cc:1261] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Refusing update from remote peer 395e8593d8cd4563bae0a5e5c3e725c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:10.355479 26846 raft_consensus.cc:1261] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Refusing update from remote peer 395e8593d8cd4563bae0a5e5c3e725c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:10.355643 27239 consensus_queue.cc:1035] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:10.355753 27199 consensus_queue.cc:1035] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:10.356317 27203 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: TxnStatusTablet state changed. Reason: New leader 395e8593d8cd4563bae0a5e5c3e725c3. Latest consensus state: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.356405 27203 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:10.356899 27215 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: TxnStatusTablet state changed. Reason: New leader 395e8593d8cd4563bae0a5e5c3e725c3. Latest consensus state: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.356997 27215 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:10.357384 27199 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.357471 27199 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:10.357446 27239 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.357517 27239 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 395e8593d8cd4563bae0a5e5c3e725c3: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:10.357584 27244 mvcc.cc:205] Tried to move back new op lower bound from 7298839553453441024 to 7298839553403011072. Current Snapshot: MvccSnapshot[applied={T|T < 7298839553453441024}]
I20260620 08:14:10.357949 27203 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.358024 27203 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 045d9e8640bd4df49ee686c86ebdd93c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:10.358022 27215 tablet_replica.cc:439] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 } } peers { permanent_uuid: "045d9e8640bd4df49ee686c86ebdd93c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46357 } } peers { permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 } } }
I20260620 08:14:10.358104 27215 tablet_replica.cc:442] T 29106fecffeb4bf3bcc8f225351e7b23 P 0b89c546db554ceab398a5634669cb38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:10.362721 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.360164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.24.249.131:46357)
0620 08:14:10.360203 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.360205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.360243 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:14:10.360328 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.360328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.360343 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.360394 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:10.360511 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.360956 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.362055 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.362253 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.362309 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.362461 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:10.362553 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:10.362602 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.362659 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.362673 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:10.366079 27246 mvcc.cc:205] Tried to move back new op lower bound from 7298839553453441024 to 7298839553403011072. Current Snapshot: MvccSnapshot[applied={T|T < 7298839553453441024}]
I20260620 08:14:10.368211 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.365252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:59773 (local address 127.24.249.130:46243)
0620 08:14:10.365297 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.365299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.365344 (+ 45us) server_negotiation.cc:407] Connection header received
0620 08:14:10.365418 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.365419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.365427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.365478 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:10.365691 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.366405 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.367524 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.367748 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.367809 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.367999 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:10.368025 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:10.368063 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.368137 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.368154 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:14:10.368270 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.365132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46243 (local address 127.24.249.131:59773)
0620 08:14:10.365291 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.365301 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.365358 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.365496 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.365497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.365498 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:10.365551 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.365552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.366486 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.366487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.367437 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.367439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.367832 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.367833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.367904 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.367922 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:10.368079 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:10.368080 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.368134 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.368180 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260620 08:14:10.370837 26986 raft_consensus.cc:1261] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Refusing update from remote peer 045d9e8640bd4df49ee686c86ebdd93c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:10.370848 26846 raft_consensus.cc:1261] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Refusing update from remote peer 045d9e8640bd4df49ee686c86ebdd93c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:10.371061 27228 consensus_queue.cc:1035] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [LEADER]: Connected to new peer: Peer: permanent_uuid: "395e8593d8cd4563bae0a5e5c3e725c3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:10.371212 27203 consensus_queue.cc:1035] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b89c546db554ceab398a5634669cb38" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:10.379864 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.377005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:42723 (local address 127.24.249.131:46357)
0620 08:14:10.377108 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.377110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.377172 (+ 62us) server_negotiation.cc:407] Connection header received
0620 08:14:10.377243 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.377244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.377251 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.377290 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:10.377424 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.377957 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.379081 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.379296 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.379353 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.379585 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:10.379616 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:10.379672 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.379743 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.379762 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 08:14:10.380028 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.376964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46357 (local address 127.24.249.130:42723)
0620 08:14:10.377114 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:10.377126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:10.377187 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:10.377303 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:10.377304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:10.377305 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:10.377367 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.377369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.378025 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.378027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.378988 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:10.378991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.379392 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.379394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:10.379462 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.379480 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:10.379686 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:10.379687 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:14:10.379742 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:14:10.379786 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260620 08:14:10.389213 26688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:10.389343 26688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:10.389984 26688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 791985de669f4265a2faf2dfce498f4c: Sending DeleteTablet for 3 replicas of tablet dcf08c25a82047d4b40761b6800247f3
I20260620 08:14:10.390291 26963 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf08c25a82047d4b40761b6800247f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:10 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:55310
I20260620 08:14:10.390311 26826 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf08c25a82047d4b40761b6800247f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:10 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:35466
I20260620 08:14:10.390535 27093 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf08c25a82047d4b40761b6800247f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:10 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:47852
I20260620 08:14:10.390724 27267 tablet_replica.cc:330] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: stopping tablet replica
I20260620 08:14:10.390923 27267 raft_consensus.cc:2229] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:10.391040 27267 raft_consensus.cc:2258] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:10.391350 27267 ts_tablet_manager.cc:1905] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:10.393416 27267 ts_tablet_manager.cc:1918] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:10.393492 27267 log.cc:1196] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/dcf08c25a82047d4b40761b6800247f3
I20260620 08:14:10.393793 27267 ts_tablet_manager.cc:1939] T dcf08c25a82047d4b40761b6800247f3 P 045d9e8640bd4df49ee686c86ebdd93c: Deleting consensus metadata
I20260620 08:14:10.394217 26676 catalog_manager.cc:4841] TS 045d9e8640bd4df49ee686c86ebdd93c (127.24.249.131:46357): tablet dcf08c25a82047d4b40761b6800247f3 (table test-table [id=fecec18b6f9e4bdaa5408b5c04ea8044]) successfully deleted
I20260620 08:14:10.397951 27266 tablet_replica.cc:330] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: stopping tablet replica
I20260620 08:14:10.398023 27265 tablet_replica.cc:330] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: stopping tablet replica
I20260620 08:14:10.398046 27266 raft_consensus.cc:2229] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:10.398098 27266 raft_consensus.cc:2258] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:10.398106 27265 raft_consensus.cc:2229] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:10.398154 27265 raft_consensus.cc:2258] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:10.398435 27266 ts_tablet_manager.cc:1905] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:10.400240 27266 ts_tablet_manager.cc:1918] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:10.400305 27266 log.cc:1196] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/dcf08c25a82047d4b40761b6800247f3
I20260620 08:14:10.400548 27266 ts_tablet_manager.cc:1939] T dcf08c25a82047d4b40761b6800247f3 P 0b89c546db554ceab398a5634669cb38: Deleting consensus metadata
I20260620 08:14:10.400976 26676 catalog_manager.cc:4841] TS 0b89c546db554ceab398a5634669cb38 (127.24.249.129:44781): tablet dcf08c25a82047d4b40761b6800247f3 (table test-table [id=fecec18b6f9e4bdaa5408b5c04ea8044]) successfully deleted
I20260620 08:14:10.402326 27265 ts_tablet_manager.cc:1905] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:10.404424 27265 ts_tablet_manager.cc:1918] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:10.404484 27265 log.cc:1196] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/dcf08c25a82047d4b40761b6800247f3
I20260620 08:14:10.404781 27265 ts_tablet_manager.cc:1939] T dcf08c25a82047d4b40761b6800247f3 P 395e8593d8cd4563bae0a5e5c3e725c3: Deleting consensus metadata
I20260620 08:14:10.405189 26676 catalog_manager.cc:4841] TS 395e8593d8cd4563bae0a5e5c3e725c3 (127.24.249.130:46243): tablet dcf08c25a82047d4b40761b6800247f3 (table test-table [id=fecec18b6f9e4bdaa5408b5c04ea8044]) successfully deleted
Jun 20 08:14:10 dist-test-slave-65w8 krb5kdc[26644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943249, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:10.420647 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.391474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.24.249.129:44781)
0620 08:14:10.391522 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.391524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.410081 (+ 18557us) server_negotiation.cc:407] Connection header received
0620 08:14:10.410170 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.410172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.410218 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.410272 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:10.411021 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.411801 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.412884 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.413202 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.413243 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.417138 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:10.417151 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:10.417152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:10.417160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:10.419274 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:10.419703 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:10.419708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:10.419708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:10.419790 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:10.420017 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:10.420018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:10.420019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:10.420212 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:10.420358 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.420456 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.420541 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 08:14:10.421149 26886 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35484
I20260620 08:14:10.433825 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:10.424344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.24.249.190:45401)
0620 08:14:10.424402 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:10.424404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:10.426106 (+ 1702us) server_negotiation.cc:407] Connection header received
0620 08:14:10.426187 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:10.426189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:10.426202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:10.426264 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:10.426983 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.427778 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.428905 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:10.429213 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:10.429289 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:10.430866 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:10.430867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:10.430868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:10.430876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:10.432550 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:10.432961 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:10.432962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:10.432962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:10.433029 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:10.433240 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:10.433241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:10.433241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:10.433385 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:10.433486 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:10.433633 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:14:10.433721 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 08:14:10.434357 26688 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33466
I20260620 08:14:10.435544 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 26761
I20260620 08:14:10.441982 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 26901
I20260620 08:14:10.448186 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 27041
I20260620 08:14:10.452816 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 26657
2026-06-20T08:14:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2674 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:10 dist-test-slave-65w8 krb5kdc[27274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:10 dist-test-slave-65w8 krb5kdc[27274](info): set up 2 sockets
Jun 20 08:14:10 dist-test-slave-65w8 krb5kdc[27274](info): commencing operation
krb5kdc: starting...
W20260620 08:14:11.497745 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:11 dist-test-slave-65w8 krb5kdc[27274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:11.625826 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:39631
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:11.711114 27287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:11.711370 27287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:11.711409 27287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:11.712654 27287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:11.712728 27287 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:14:11.712749 27287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:11.712770 27287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:11.712803 27287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:11.712831 27287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:11.714437 27287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:39631
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:11.714737 27287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:11.714974 27287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:14:11.720405 27287 server_base.cc:1034] running on GCE node
W20260620 08:14:11.720366 27293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:11.720366 27292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:11.720366 27295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:11.721191 27287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:11.721678 27287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:11.722854 27287 hybrid_clock.cc:648] HybridClock initialized: now 1781943251722844 us; error 37 us; skew 500 ppm
Jun 20 08:14:11 dist-test-slave-65w8 krb5kdc[27274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943251, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:11.725660 27287 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:11.726338 27287 webserver.cc:460] Webserver started at http://127.24.249.190:38365/ using document root <none> and password file <none>
I20260620 08:14:11.726577 27287 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:11.726643 27287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:11.726805 27287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:11.727960 27287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "2d9f6bd3180540c18cb6c467b4677c70"
format_stamp: "Formatted at 2026-06-20 08:14:11 on dist-test-slave-65w8"
server_key: "d9b604f81b484c8a59b377c918e1356e"
server_key_iv: "a0748f81f481af3a8f9472769e3d7409"
server_key_version: "encryptionkey@0"
I20260620 08:14:11.728334 27287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "2d9f6bd3180540c18cb6c467b4677c70"
format_stamp: "Formatted at 2026-06-20 08:14:11 on dist-test-slave-65w8"
server_key: "d9b604f81b484c8a59b377c918e1356e"
server_key_iv: "a0748f81f481af3a8f9472769e3d7409"
server_key_version: "encryptionkey@0"
I20260620 08:14:11.729929 27287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:11.730849 27302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:11.731030 27287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:11.731132 27287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "2d9f6bd3180540c18cb6c467b4677c70"
format_stamp: "Formatted at 2026-06-20 08:14:11 on dist-test-slave-65w8"
server_key: "d9b604f81b484c8a59b377c918e1356e"
server_key_iv: "a0748f81f481af3a8f9472769e3d7409"
server_key_version: "encryptionkey@0"
I20260620 08:14:11.731216 27287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:11.741426 27287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:11.744236 27287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:11.744439 27287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:11.749342 27287 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:39631
I20260620 08:14:11.749387 27364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:39631 every 8 connection(s)
I20260620 08:14:11.749729 27287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:11.750451 27365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:11.751240 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 27287
I20260620 08:14:11.751333 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:11.751507 25574 external_mini_cluster.cc:1428] Setting key f39c2ed2316266a073995de332cb1f44
I20260620 08:14:11.753556 27365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Bootstrap starting.
I20260620 08:14:11.754218 27365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:11.754515 27365 log.cc:824] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:11.755347 27365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: No bootstrap required, opened a new log
I20260620 08:14:11.757959 27365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9f6bd3180540c18cb6c467b4677c70" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39631 } }
I20260620 08:14:11.758175 27365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:11.758231 27365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9f6bd3180540c18cb6c467b4677c70, State: Initialized, Role: FOLLOWER
I20260620 08:14:11.758384 27365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9f6bd3180540c18cb6c467b4677c70" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39631 } }
I20260620 08:14:11.758508 27365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:11.758548 27365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:11.758590 27365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:14:11 dist-test-slave-65w8 krb5kdc[27274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:11.759210 27365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9f6bd3180540c18cb6c467b4677c70" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39631 } }
I20260620 08:14:11.759297 27365 leader_election.cc:302] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d9f6bd3180540c18cb6c467b4677c70; no voters:
I20260620 08:14:11.759498 27365 leader_election.cc:288] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:11.759542 27370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:11.759694 27370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [term 1 LEADER]: Becoming Leader. State: Replica: 2d9f6bd3180540c18cb6c467b4677c70, State: Running, Role: LEADER
I20260620 08:14:11.759765 27365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:11.759791 27370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9f6bd3180540c18cb6c467b4677c70" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39631 } }
I20260620 08:14:11.760166 27370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d9f6bd3180540c18cb6c467b4677c70. Latest consensus state: current_term: 1 leader_uuid: "2d9f6bd3180540c18cb6c467b4677c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9f6bd3180540c18cb6c467b4677c70" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39631 } } }
I20260620 08:14:11.760257 27370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:11.760471 27371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d9f6bd3180540c18cb6c467b4677c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9f6bd3180540c18cb6c467b4677c70" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39631 } } }
I20260620 08:14:11.760552 27371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:11.761302 27377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:11.761981 27377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:11.763363 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:11.752124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.24.249.190:39631)
0620 08:14:11.752324 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:11.752333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:11.752359 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:14:11.752772 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:11.752778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:11.752827 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:11.752980 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:11.753603 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.754685 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.755889 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.756328 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.756420 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:11.759386 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:11.759391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:11.759391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:11.759405 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:11.761662 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:11.762118 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:11.762121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:11.762122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:11.762212 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:11.762419 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:11.762420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:11.762420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:11.762708 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:11.762848 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:11.763028 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 08:14:11.763201 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260620 08:14:11.763756 27377 catalog_manager.cc:1347] Generated new cluster ID: aa26c0326c1248bca9947c83e11c6eb4
I20260620 08:14:11.763808 27377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:11.804143 27377 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:11.804790 27377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:11.808144 27377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Generated new TSK 0
I20260620 08:14:11.808350 27377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:11.844313 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39631
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:11.926693 27391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:11.926898 27391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:11.926923 27391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:11.928016 27391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:11.928051 27391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:11.928071 27391 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:14:11.928083 27391 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:11.928100 27391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:11.929575 27391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:11.929765 27391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:11.929965 27391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:11.935325 27396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:11.935325 27399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:11.935438 27397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:11.935546 27391 server_base.cc:1034] running on GCE node
I20260620 08:14:11.935742 27391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:11.936012 27391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:11.937237 27391 hybrid_clock.cc:648] HybridClock initialized: now 1781943251937243 us; error 49 us; skew 500 ppm
Jun 20 08:14:11 dist-test-slave-65w8 krb5kdc[27274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943251, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:11.940107 27391 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:11.940601 27391 webserver.cc:460] Webserver started at http://127.24.249.129:39713/ using document root <none> and password file <none>
I20260620 08:14:11.940865 27391 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:11.940934 27391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:11.941075 27391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:11.942252 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "8097dd42a4f54fb6b3256f292807d5ab"
format_stamp: "Formatted at 2026-06-20 08:14:11 on dist-test-slave-65w8"
server_key: "8a0b4dd46ce9aeb47e992ea353b45d84"
server_key_iv: "fd142ebe40f8c3518d643e5c55fc7f50"
server_key_version: "encryptionkey@0"
I20260620 08:14:11.942610 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "8097dd42a4f54fb6b3256f292807d5ab"
format_stamp: "Formatted at 2026-06-20 08:14:11 on dist-test-slave-65w8"
server_key: "8a0b4dd46ce9aeb47e992ea353b45d84"
server_key_iv: "fd142ebe40f8c3518d643e5c55fc7f50"
server_key_version: "encryptionkey@0"
I20260620 08:14:11.944042 27391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:11.944954 27406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:11.945232 27391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:11.945353 27391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "8097dd42a4f54fb6b3256f292807d5ab"
format_stamp: "Formatted at 2026-06-20 08:14:11 on dist-test-slave-65w8"
server_key: "8a0b4dd46ce9aeb47e992ea353b45d84"
server_key_iv: "fd142ebe40f8c3518d643e5c55fc7f50"
server_key_version: "encryptionkey@0"
I20260620 08:14:11.945439 27391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:11.958869 27391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:11.961239 27391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:11.961398 27391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:11.962126 27391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:11.962178 27391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:11.962210 27391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:11.962232 27391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:14:11 dist-test-slave-65w8 krb5kdc[27274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943251, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:11.969699 27391 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:39479
I20260620 08:14:11.969796 27521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:39479 every 8 connection(s)
I20260620 08:14:11.970172 27391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:11.972038 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:11.962425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:55499 (local address 127.24.249.190:39631)
0620 08:14:11.962472 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:11.962474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:11.962673 (+ 199us) server_negotiation.cc:407] Connection header received
0620 08:14:11.963212 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:11.963214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:11.963227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:11.963274 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:11.963907 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.964429 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.965456 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.965672 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.965756 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:11.969109 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:11.969111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:11.969111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:11.969120 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:11.970812 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:11.971202 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:11.971203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:11.971203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:11.971265 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:11.971451 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:11.971452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:11.971452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:11.971555 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:11.971613 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:11.971886 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 08:14:11.971973 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:14:11.972105 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:11.962410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39631 (local address 127.24.249.129:55499)
0620 08:14:11.962557 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:11.962593 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:11.963099 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:11.963364 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:11.963368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:11.963619 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:11.963812 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:11.963816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.964526 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.964527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:11.965362 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:11.965365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.965769 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.965770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:11.965940 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:11.966476 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:11.966488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:11.968988 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:11.970884 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:11.970888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:11.970889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:11.971104 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:11.971328 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:11.971329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:11.971329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:11.971415 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:11.971602 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:11.971604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:11.971744 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 08:14:11.971830 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:14:11.971951 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260620 08:14:11.972630 27522 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39631
I20260620 08:14:11.972719 27522 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:11.972888 27522 heartbeater.cc:511] Master 127.24.249.190:39631 requested a full tablet report, sending...
I20260620 08:14:11.973425 27318 ts_manager.cc:194] Registered new tserver with Master: 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479)
I20260620 08:14:11.974256 27318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:55499
I20260620 08:14:11.979533 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:11.973944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:52381 (local address 127.24.249.190:39631)
0620 08:14:11.973984 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:11.973986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:11.974019 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:14:11.974085 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:11.974086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:11.974093 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:11.974142 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:11.974636 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.975098 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.975789 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.975970 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.976021 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:11.976894 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:11.976895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:11.976896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:11.976902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:11.978475 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:11.978824 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:11.978824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:11.978825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:11.978872 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:11.979023 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:11.979024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:11.979024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:11.979115 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:11.979164 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:11.979404 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 08:14:11.979472 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:11.979568 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:11.973803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39631 (local address 127.24.249.129:52381)
0620 08:14:11.973962 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:11.973974 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:11.974033 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:11.974158 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:11.974159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:11.974362 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:11.974429 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:11.974432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.975217 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.975218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:11.975714 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:11.975715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:11.976034 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:11.976035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:11.976092 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:11.976529 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:11.976533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:11.976810 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:11.978550 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:11.978551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:11.978551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:11.978753 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:11.978924 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:11.978924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:11.978925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:11.978975 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:11.979213 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:11.979214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:11.979255 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 08:14:11.979323 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:14:11.979459 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260620 08:14:11.979553 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 27391
I20260620 08:14:11.979669 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:11.979875 25574 external_mini_cluster.cc:1428] Setting key a02167fe46c3849e54b30489799e77ae
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:12.010308 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39631
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:12.093258 27531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:12.093478 27531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:12.093503 27531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:12.094748 27531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:12.094790 27531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:12.094811 27531 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:14:12.094823 27531 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:12.094841 27531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:12.096294 27531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:12.096549 27531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:12.096807 27531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:12.101740 27536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:12.101740 27537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:12.101905 27531 server_base.cc:1034] running on GCE node
W20260620 08:14:12.101722 27539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:12.102156 27531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:12.102396 27531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:12.103550 27531 hybrid_clock.cc:648] HybridClock initialized: now 1781943252103533 us; error 28 us; skew 500 ppm
Jun 20 08:14:12 dist-test-slave-65w8 krb5kdc[27274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943252, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:12.106137 27531 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:12.106662 27531 webserver.cc:460] Webserver started at http://127.24.249.130:45523/ using document root <none> and password file <none>
I20260620 08:14:12.106940 27531 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:12.107009 27531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:12.107169 27531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:12.108189 27531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "b247f2f5566c4eb19503a9bfa694647c"
format_stamp: "Formatted at 2026-06-20 08:14:12 on dist-test-slave-65w8"
server_key: "419e19190df75d710f3a0027f4f72495"
server_key_iv: "36e7f6f4455229d894c189e1664b0047"
server_key_version: "encryptionkey@0"
I20260620 08:14:12.108481 27531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "b247f2f5566c4eb19503a9bfa694647c"
format_stamp: "Formatted at 2026-06-20 08:14:12 on dist-test-slave-65w8"
server_key: "419e19190df75d710f3a0027f4f72495"
server_key_iv: "36e7f6f4455229d894c189e1664b0047"
server_key_version: "encryptionkey@0"
I20260620 08:14:12.109709 27531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:14:12.110515 27546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:12.110721 27531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:12.110832 27531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "b247f2f5566c4eb19503a9bfa694647c"
format_stamp: "Formatted at 2026-06-20 08:14:12 on dist-test-slave-65w8"
server_key: "419e19190df75d710f3a0027f4f72495"
server_key_iv: "36e7f6f4455229d894c189e1664b0047"
server_key_version: "encryptionkey@0"
I20260620 08:14:12.110908 27531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:12.121481 27531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:12.123854 27531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:12.123996 27531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:12.124616 27531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:12.124665 27531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:12.124697 27531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:12.124717 27531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:14:12 dist-test-slave-65w8 krb5kdc[27274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943252, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:12.131830 27531 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:40655
I20260620 08:14:12.131906 27661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:40655 every 8 connection(s)
I20260620 08:14:12.132251 27531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:12.134732 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.125008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:52715 (local address 127.24.249.190:39631)
0620 08:14:12.125086 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.125089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.125324 (+ 235us) server_negotiation.cc:407] Connection header received
0620 08:14:12.125797 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.125799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.125814 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.125928 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:12.126570 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.127426 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.128529 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.128733 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.128826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.131658 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:12.131660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:12.131660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:12.131670 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:12.133274 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.133754 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.133755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.133755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.133813 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.134079 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.134080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.134081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.134208 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:12.134283 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.134542 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.134644 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:14:12.134904 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.125004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39631 (local address 127.24.249.130:52715)
0620 08:14:12.125167 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.125207 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.125705 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.126008 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.126012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.126311 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:12.126495 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.126500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.127525 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.127525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.128426 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.128428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.128834 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.128835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.128972 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.129507 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:12.129525 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:12.131551 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:12.133376 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.133382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.133384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.133665 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.133906 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.133907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.133907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.134014 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.134273 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:12.134280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:12.134417 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.134523 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.134686 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260620 08:14:12.135403 27662 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39631
I20260620 08:14:12.135509 27662 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:12.135564 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 27531
I20260620 08:14:12.135665 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:12.135721 27662 heartbeater.cc:511] Master 127.24.249.190:39631 requested a full tablet report, sending...
I20260620 08:14:12.135900 25574 external_mini_cluster.cc:1428] Setting key 6bb4333327dd775b25102a0ddedd0ebf
I20260620 08:14:12.136122 27318 ts_manager.cc:194] Registered new tserver with Master: b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655)
I20260620 08:14:12.136655 27318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:52715
I20260620 08:14:12.142561 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.136402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:38759 (local address 127.24.249.190:39631)
0620 08:14:12.136436 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.136438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.136538 (+ 100us) server_negotiation.cc:407] Connection header received
0620 08:14:12.136606 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.136606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.136622 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.136680 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:12.137037 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.137491 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.138580 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.138934 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.139009 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.140032 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:12.140033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:12.140033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:12.140039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:12.141378 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.141793 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.141794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.141794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.141844 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.142090 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.142091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.142091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.142227 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:12.142282 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.142443 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.142503 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:12.142652 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.136344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39631 (local address 127.24.249.130:38759)
0620 08:14:12.136472 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.136484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.136541 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.136694 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.136695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.136906 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:12.136970 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.136972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.137583 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.137584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.138494 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.138496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.139034 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.139035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.139121 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.139494 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:12.139500 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:12.139950 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:12.141454 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.141455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.141456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.141697 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.141952 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.141953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.141954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.142025 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.142294 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:12.142295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:12.142341 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.142426 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.142528 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:12.162853 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39631
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:12.246742 27671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:12.246953 27671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:12.246977 27671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:12.248152 27671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:12.248195 27671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:12.248216 27671 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 08:14:12.248229 27671 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:12.248262 27671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:12.249936 27671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:12.250167 27671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:12.250406 27671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:12.255380 27676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:12.255345 27679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:12.255470 27671 server_base.cc:1034] running on GCE node
W20260620 08:14:12.255353 27677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:12.255896 27671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:12.256160 27671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:12.257339 27671 hybrid_clock.cc:648] HybridClock initialized: now 1781943252257320 us; error 35 us; skew 500 ppm
Jun 20 08:14:12 dist-test-slave-65w8 krb5kdc[27274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943252, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:12.260181 27671 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:12.260798 27671 webserver.cc:460] Webserver started at http://127.24.249.131:35553/ using document root <none> and password file <none>
I20260620 08:14:12.261034 27671 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:12.261129 27671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:12.261276 27671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:12.262426 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "2e345d485c554efa8811e04f95f43141"
format_stamp: "Formatted at 2026-06-20 08:14:12 on dist-test-slave-65w8"
server_key: "d9c6469808592c2fd1bdc64488c1394f"
server_key_iv: "b2107520cefac4bbc24738302db47566"
server_key_version: "encryptionkey@0"
I20260620 08:14:12.262785 27671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "2e345d485c554efa8811e04f95f43141"
format_stamp: "Formatted at 2026-06-20 08:14:12 on dist-test-slave-65w8"
server_key: "d9c6469808592c2fd1bdc64488c1394f"
server_key_iv: "b2107520cefac4bbc24738302db47566"
server_key_version: "encryptionkey@0"
I20260620 08:14:12.264359 27671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:12.265215 27686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:12.265403 27671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:12.265507 27671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "2e345d485c554efa8811e04f95f43141"
format_stamp: "Formatted at 2026-06-20 08:14:12 on dist-test-slave-65w8"
server_key: "d9c6469808592c2fd1bdc64488c1394f"
server_key_iv: "b2107520cefac4bbc24738302db47566"
server_key_version: "encryptionkey@0"
I20260620 08:14:12.265594 27671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:12.314170 27671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:12.316562 27671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:12.316727 27671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:12.317268 27671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:12.317332 27671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:12.317368 27671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:12.317409 27671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:12.324131 27671 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:35687
I20260620 08:14:12.324178 27801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:35687 every 8 connection(s)
I20260620 08:14:12.324497 27671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:14:12 dist-test-slave-65w8 krb5kdc[27274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943252, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:12.327804 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.317771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:40177 (local address 127.24.249.190:39631)
0620 08:14:12.317815 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.317818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.318105 (+ 287us) server_negotiation.cc:407] Connection header received
0620 08:14:12.318667 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.318668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.318681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.318771 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:12.319649 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.320520 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.322098 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.322420 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.322518 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.325169 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:12.325171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:12.325171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:12.325178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:12.326441 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.326851 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.326852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.326852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.326912 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.327175 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.327176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.327176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.327298 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:12.327363 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.327632 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.327740 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:12.327907 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.317747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39631 (local address 127.24.249.131:40177)
0620 08:14:12.317959 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.317994 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.318565 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.318854 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.318862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.319250 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:12.319537 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.319543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.320618 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.320619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.322022 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.322025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.322530 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.322530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.322641 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.323255 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:12.323267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:12.325083 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:12.326516 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.326522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.326524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.326755 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.326980 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.326980 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.326981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.327093 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.327369 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:12.327374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:12.327488 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.327578 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.327703 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260620 08:14:12.328454 27802 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39631
I20260620 08:14:12.328541 27802 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:12.328742 27802 heartbeater.cc:511] Master 127.24.249.190:39631 requested a full tablet report, sending...
I20260620 08:14:12.328825 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 27671
I20260620 08:14:12.328910 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:12.329116 25574 external_mini_cluster.cc:1428] Setting key f3ec6cb222730605fb97ec6ea2eb1365
I20260620 08:14:12.329131 27319 ts_manager.cc:194] Registered new tserver with Master: 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:12.329654 27319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:40177
I20260620 08:14:12.329950 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:14:12.335870 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.329456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39631 (local address 127.24.249.131:38831)
0620 08:14:12.329635 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.329647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.329704 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.329948 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.329949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.330135 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:12.330188 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.330190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.330811 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.330812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.331364 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.331365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.331810 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.331811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.331885 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.332333 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:12.332337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:12.332610 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:12.334600 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.334600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.334601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.334847 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.335070 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.335071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.335072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.335135 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.335511 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:12.335512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:12.335575 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.335667 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.335792 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260620 08:14:12.335978 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.329486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:38831 (local address 127.24.249.190:39631)
0620 08:14:12.329523 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.329525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.329690 (+ 165us) server_negotiation.cc:407] Connection header received
0620 08:14:12.329850 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.329851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.329882 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.329934 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:12.330268 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.330727 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.331444 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.331729 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.331799 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.332711 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:12.332713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:12.332714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:12.332721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:12.334520 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.334920 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.334921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.334922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.334985 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.335246 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.335247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.335248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.335431 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:12.335517 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.335705 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.335833 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 08:14:12 dist-test-slave-65w8 krb5kdc[27274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:14:12 dist-test-slave-65w8 krb5kdc[27274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:12.354538 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.347469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.24.249.190:39631)
0620 08:14:12.347562 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.347564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.347575 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:12.347601 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.347602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.347616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.347668 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:12.348253 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.348857 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.349633 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.349824 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.349922 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.352134 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:12.352135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:12.352135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:12.352142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:12.353516 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.353849 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.353849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.353850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.353964 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.354131 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.354132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.354132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.354257 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:12.354332 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.354411 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.354471 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:14:12.356407 27318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:12.358098 27318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:12.364786 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.360572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.24.249.130:40655)
0620 08:14:12.360982 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.360986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.360995 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:12.361030 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.361033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.361047 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.361176 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:12.361401 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.362279 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.363807 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.364322 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.364406 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.364462 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.364655 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.364693 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":89,"threads_started":1}
I20260620 08:14:12.364820 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.360850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.24.249.131:35687)
0620 08:14:12.361019 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.361023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.361030 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:12.361082 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.361084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.361100 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.361172 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:12.361443 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.362593 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.363655 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.364327 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.364400 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.364455 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.364664 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.364703 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260620 08:14:12.364775 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.360460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:40655 (local address 127.0.0.1:60976)
0620 08:14:12.360641 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.360672 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.360773 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.361176 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.361177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.361179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:12.361242 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.361244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.362369 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.362370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.363706 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.363709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.364414 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.364415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.364502 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.364521 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.364612 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.364684 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":65,"thread_start_us":60,"threads_started":1}
I20260620 08:14:12.365060 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.360776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.24.249.129:39479)
0620 08:14:12.360934 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.360940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.360950 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:12.360992 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.360994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.361011 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.361093 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:12.361282 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.362571 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.363866 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.364574 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.364845 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.364907 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.364931 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.364970 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260620 08:14:12.365010 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.360460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39479 (local address 127.0.0.1:56462)
0620 08:14:12.360622 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.360723 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.360773 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.361106 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.361108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.361111 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:12.361194 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.361196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.362659 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.362660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.363789 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.363791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.364659 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.364660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.364721 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.364724 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.364755 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.364795 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":33,"thread_start_us":89,"threads_started":1}
I20260620 08:14:12.365345 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.360659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35687 (local address 127.0.0.1:58252)
0620 08:14:12.360873 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.360883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.360922 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.361261 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.361262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.361264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:12.361361 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.361363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.362676 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.362677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.363577 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.363579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.364490 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.364491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.364564 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.364568 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.364619 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.365281 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":614,"thread_start_us":72,"threads_started":1}
I20260620 08:14:12.365800 27596 tablet_service.cc:1467] Processing CreateTablet for tablet c18f7d69417348ddb831025dde914905 (DEFAULT_TABLE table=test-table [id=d8a1e1ce722f44dbab6997e4d40bc1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:12.365911 27456 tablet_service.cc:1467] Processing CreateTablet for tablet c18f7d69417348ddb831025dde914905 (DEFAULT_TABLE table=test-table [id=d8a1e1ce722f44dbab6997e4d40bc1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:12.366250 27596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18f7d69417348ddb831025dde914905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.366264 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18f7d69417348ddb831025dde914905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.366397 27736 tablet_service.cc:1467] Processing CreateTablet for tablet c18f7d69417348ddb831025dde914905 (DEFAULT_TABLE table=test-table [id=d8a1e1ce722f44dbab6997e4d40bc1c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:12.366724 27736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18f7d69417348ddb831025dde914905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.369490 27825 tablet_bootstrap.cc:492] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Bootstrap starting.
I20260620 08:14:12.370016 27827 tablet_bootstrap.cc:492] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Bootstrap starting.
I20260620 08:14:12.370136 27826 tablet_bootstrap.cc:492] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Bootstrap starting.
I20260620 08:14:12.370210 27825 tablet_bootstrap.cc:654] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.370563 27825 log.cc:824] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:12.370695 27827 tablet_bootstrap.cc:654] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.370992 27827 log.cc:824] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:12.370994 27826 tablet_bootstrap.cc:654] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.371330 27826 log.cc:824] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:12.371486 27825 tablet_bootstrap.cc:492] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: No bootstrap required, opened a new log
I20260620 08:14:12.371557 27825 ts_tablet_manager.cc:1397] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:12.371798 27827 tablet_bootstrap.cc:492] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: No bootstrap required, opened a new log
I20260620 08:14:12.372057 27826 tablet_bootstrap.cc:492] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: No bootstrap required, opened a new log
I20260620 08:14:12.372121 27826 ts_tablet_manager.cc:1397] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:12.372229 27827 ts_tablet_manager.cc:1397] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:12.373683 27825 raft_consensus.cc:348] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.373888 27825 raft_consensus.cc:374] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.373775 27826 raft_consensus.cc:348] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.373939 27825 raft_consensus.cc:729] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b247f2f5566c4eb19503a9bfa694647c, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.373947 27826 raft_consensus.cc:374] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.373982 27826 raft_consensus.cc:729] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e345d485c554efa8811e04f95f43141, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.374104 27826 consensus_queue.cc:260] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.374122 27825 consensus_queue.cc:260] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.374380 27825 ts_tablet_manager.cc:1428] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 08:14:12.374393 27802 heartbeater.cc:503] Master 127.24.249.190:39631 was elected leader, sending a full tablet report...
I20260620 08:14:12.374377 27826 ts_tablet_manager.cc:1428] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:14:12.374655 27662 heartbeater.cc:503] Master 127.24.249.190:39631 was elected leader, sending a full tablet report...
I20260620 08:14:12.374326 27827 raft_consensus.cc:348] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.375037 27827 raft_consensus.cc:374] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.375065 27827 raft_consensus.cc:729] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8097dd42a4f54fb6b3256f292807d5ab, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.375140 27827 consensus_queue.cc:260] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.375412 27522 heartbeater.cc:503] Master 127.24.249.190:39631 was elected leader, sending a full tablet report...
I20260620 08:14:12.375443 27827 ts_tablet_manager.cc:1428] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 08:14:12.383294 27663 tablet.cc:2369] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:14:12.470983 27523 tablet.cc:2369] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:12.473670 27833 raft_consensus.cc:482] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:12.473850 27833 raft_consensus.cc:504] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.474287 27833 leader_election.cc:288] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:12.477651 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.474904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:33425 (local address 127.24.249.131:35687)
0620 08:14:12.474969 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.474971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.474982 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:12.475022 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.475022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.475031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.475090 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:12.475247 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.475695 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.476789 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.477316 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.477394 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.477402 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.477543 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.477587 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:14:12.477667 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.474758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35687 (local address 127.24.249.129:33425)
0620 08:14:12.474872 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.474885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.474960 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.475091 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.475092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.475093 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:12.475173 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.475175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.475782 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.475783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.476665 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.476667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.477404 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.477405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.477463 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.477467 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.477517 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.477547 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":37,"thread_start_us":47,"threads_started":1}
I20260620 08:14:12.477793 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.474712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:46037 (local address 127.24.249.130:40655)
0620 08:14:12.474898 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.474900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.474913 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:14:12.474943 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.474944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.474956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.475017 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:12.475180 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.475696 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.476753 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.477325 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.477368 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.477379 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.477611 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.477656 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260620 08:14:12.477763 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.474570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:40655 (local address 127.24.249.129:46037)
0620 08:14:12.474731 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.474745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.474807 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.475031 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.475032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.475035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:12.475110 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.475113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.475780 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.475781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.476652 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.476655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.477441 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.477442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.477530 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.477535 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.477587 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.477626 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":49,"thread_start_us":67,"threads_started":1}
I20260620 08:14:12.478250 27616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18f7d69417348ddb831025dde914905" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b247f2f5566c4eb19503a9bfa694647c" is_pre_election: true
I20260620 08:14:12.478245 27756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18f7d69417348ddb831025dde914905" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141" is_pre_election: true
I20260620 08:14:12.478401 27616 raft_consensus.cc:2454] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 0.
I20260620 08:14:12.478410 27756 raft_consensus.cc:2454] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 0.
I20260620 08:14:12.478629 27408 leader_election.cc:302] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e345d485c554efa8811e04f95f43141, 8097dd42a4f54fb6b3256f292807d5ab; no voters:
I20260620 08:14:12.478755 27833 raft_consensus.cc:2781] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:12.478811 27833 raft_consensus.cc:482] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:12.478842 27833 raft_consensus.cc:3037] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.479669 27833 raft_consensus.cc:504] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.479831 27833 leader_election.cc:288] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 election: Requested vote from peers b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:12.480005 27616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18f7d69417348ddb831025dde914905" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b247f2f5566c4eb19503a9bfa694647c"
I20260620 08:14:12.480017 27756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18f7d69417348ddb831025dde914905" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141"
I20260620 08:14:12.480082 27616 raft_consensus.cc:3037] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.480101 27756 raft_consensus.cc:3037] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.480767 27616 raft_consensus.cc:2454] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 1.
I20260620 08:14:12.480947 27408 leader_election.cc:302] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8097dd42a4f54fb6b3256f292807d5ab, b247f2f5566c4eb19503a9bfa694647c; no voters:
I20260620 08:14:12.481046 27833 raft_consensus.cc:2781] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:12.481081 27756 raft_consensus.cc:2454] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 1.
I20260620 08:14:12.481192 27833 raft_consensus.cc:686] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 LEADER]: Becoming Leader. State: Replica: 8097dd42a4f54fb6b3256f292807d5ab, State: Running, Role: LEADER
I20260620 08:14:12.481295 27833 consensus_queue.cc:237] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.482339 27317 catalog_manager.cc:5495] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129). New cstate: current_term: 1 leader_uuid: "8097dd42a4f54fb6b3256f292807d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } health_report { overall_health: UNKNOWN } } }
Jun 20 08:14:12 dist-test-slave-65w8 krb5kdc[27274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943251, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:12.534893 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.526803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.24.249.190:39631)
0620 08:14:12.526963 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.526966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.527085 (+ 119us) server_negotiation.cc:407] Connection header received
0620 08:14:12.527175 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.527176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.527187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.527256 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:12.527732 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.528252 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.529021 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.529203 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.529300 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.531662 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:12.531663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:12.531664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:12.531671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:12.533258 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.533683 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.533684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.533685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.533749 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:12.533987 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:12.533988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:12.533988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:12.534183 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:12.534426 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.534661 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.534774 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260620 08:14:12.534982 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.526706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39631 (local address 127.0.0.1:44566)
0620 08:14:12.527008 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.527018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.527107 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.527261 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.527263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.527591 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:12.527637 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.527639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.528349 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.528350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.528942 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.528943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.529294 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.529295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.529344 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.529998 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:12.530003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:12.531568 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:12.533381 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.533389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.533392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.533611 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.533813 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:12.533814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:12.533815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:12.533915 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:12.534410 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:12.534418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:12.534475 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.534578 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.534846 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":89,"thread_start_us":148,"threads_started":1}
I20260620 08:14:12.537930 27317 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:44566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:14:12.538153 27317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:12.540153 27456 tablet_service.cc:1467] Processing CreateTablet for tablet d50cecb4f56642809e1accc73059a4ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b17d47240ca84a6a965740452a049dc3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:12.540131 27596 tablet_service.cc:1467] Processing CreateTablet for tablet d50cecb4f56642809e1accc73059a4ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b17d47240ca84a6a965740452a049dc3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:12.540172 27736 tablet_service.cc:1467] Processing CreateTablet for tablet d50cecb4f56642809e1accc73059a4ec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b17d47240ca84a6a965740452a049dc3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:12.540341 27596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50cecb4f56642809e1accc73059a4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.540340 27736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50cecb4f56642809e1accc73059a4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.540341 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50cecb4f56642809e1accc73059a4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.541541 27827 tablet_bootstrap.cc:492] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: Bootstrap starting.
I20260620 08:14:12.542008 27827 tablet_bootstrap.cc:654] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.542245 27825 tablet_bootstrap.cc:492] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: Bootstrap starting.
I20260620 08:14:12.542259 27826 tablet_bootstrap.cc:492] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: Bootstrap starting.
I20260620 08:14:12.542821 27826 tablet_bootstrap.cc:654] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.542845 27825 tablet_bootstrap.cc:654] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.543025 27827 tablet_bootstrap.cc:492] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: No bootstrap required, opened a new log
I20260620 08:14:12.543080 27827 ts_tablet_manager.cc:1397] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:12.543236 27827 raft_consensus.cc:348] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.543322 27827 raft_consensus.cc:374] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.543347 27827 raft_consensus.cc:729] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8097dd42a4f54fb6b3256f292807d5ab, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.543401 27827 consensus_queue.cc:260] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.543495 27827 ts_tablet_manager.cc:1428] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:12.543635 27833 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.543709 27833 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:12.543803 27825 tablet_bootstrap.cc:492] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: No bootstrap required, opened a new log
I20260620 08:14:12.543855 27825 ts_tablet_manager.cc:1397] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:12.544001 27825 raft_consensus.cc:348] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.544072 27825 raft_consensus.cc:374] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.544106 27825 raft_consensus.cc:729] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b247f2f5566c4eb19503a9bfa694647c, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.544113 27826 tablet_bootstrap.cc:492] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: No bootstrap required, opened a new log
I20260620 08:14:12.544157 27826 ts_tablet_manager.cc:1397] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:12.544159 27825 consensus_queue.cc:260] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.544242 27825 ts_tablet_manager.cc:1428] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:12.544317 27826 raft_consensus.cc:348] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.544394 27826 raft_consensus.cc:374] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.544420 27826 raft_consensus.cc:729] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e345d485c554efa8811e04f95f43141, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.544480 27832 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.544566 27832 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:12.544600 27826 consensus_queue.cc:260] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.544749 27826 ts_tablet_manager.cc:1428] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:12.544934 27831 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.545038 27831 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 08:14:12.575450 27803 tablet.cc:2369] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:12.634112 27831 raft_consensus.cc:482] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:12.634239 27831 raft_consensus.cc:504] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.634718 27831 leader_election.cc:288] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479), b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655)
I20260620 08:14:12.639225 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.634954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:60071 (local address 127.24.249.130:40655)
0620 08:14:12.635003 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.635005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.635018 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:14:12.635091 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.635093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.635103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.635208 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:12.635428 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.636193 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.638110 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.638752 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.638848 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.638860 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.639094 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.639149 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:14:12.639225 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.635026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:50601 (local address 127.24.249.129:39479)
0620 08:14:12.635070 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.635072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.635206 (+ 134us) server_negotiation.cc:407] Connection header received
0620 08:14:12.635335 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.635337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.635347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.635402 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:12.635563 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.636244 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.638111 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.638696 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.638848 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.638861 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.639094 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.639151 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:12.639225 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.634893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:40655 (local address 127.24.249.131:60071)
0620 08:14:12.634940 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.634954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.635004 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.635216 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.635218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.635220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:12.635317 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.635319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.636312 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.636313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.637971 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.637973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.638901 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.638902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.638997 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.639004 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.639074 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.639134 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":175}
I20260620 08:14:12.639235 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.634940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39479 (local address 127.24.249.131:50601)
0620 08:14:12.635149 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.635160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.635229 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.635411 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.635412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.635414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:12.635487 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.635489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.636314 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.636314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.637966 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.637971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.638930 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.638931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.638997 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.639005 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.639074 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.639123 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":60,"thread_start_us":129,"threads_started":1}
I20260620 08:14:12.639608 27616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50cecb4f56642809e1accc73059a4ec" candidate_uuid: "2e345d485c554efa8811e04f95f43141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b247f2f5566c4eb19503a9bfa694647c" is_pre_election: true
I20260620 08:14:12.639724 27616 raft_consensus.cc:2454] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e345d485c554efa8811e04f95f43141 in term 0.
I20260620 08:14:12.639684 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50cecb4f56642809e1accc73059a4ec" candidate_uuid: "2e345d485c554efa8811e04f95f43141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097dd42a4f54fb6b3256f292807d5ab" is_pre_election: true
I20260620 08:14:12.639806 27476 raft_consensus.cc:2454] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e345d485c554efa8811e04f95f43141 in term 0.
I20260620 08:14:12.640038 27687 leader_election.cc:302] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e345d485c554efa8811e04f95f43141, 8097dd42a4f54fb6b3256f292807d5ab; no voters:
I20260620 08:14:12.640209 27831 raft_consensus.cc:2781] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:12.640264 27831 raft_consensus.cc:482] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:12.640304 27831 raft_consensus.cc:3037] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.641043 27831 raft_consensus.cc:504] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.641189 27831 leader_election.cc:288] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [CANDIDATE]: Term 1 election: Requested vote from peers 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479), b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655)
I20260620 08:14:12.641368 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50cecb4f56642809e1accc73059a4ec" candidate_uuid: "2e345d485c554efa8811e04f95f43141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097dd42a4f54fb6b3256f292807d5ab"
I20260620 08:14:12.641455 27476 raft_consensus.cc:3037] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.641498 27616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50cecb4f56642809e1accc73059a4ec" candidate_uuid: "2e345d485c554efa8811e04f95f43141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b247f2f5566c4eb19503a9bfa694647c"
I20260620 08:14:12.641597 27616 raft_consensus.cc:3037] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.642407 27476 raft_consensus.cc:2454] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e345d485c554efa8811e04f95f43141 in term 1.
I20260620 08:14:12.642423 27616 raft_consensus.cc:2454] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e345d485c554efa8811e04f95f43141 in term 1.
I20260620 08:14:12.642625 27688 leader_election.cc:302] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e345d485c554efa8811e04f95f43141, b247f2f5566c4eb19503a9bfa694647c; no voters:
I20260620 08:14:12.642735 27831 raft_consensus.cc:2781] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:12.642889 27831 raft_consensus.cc:686] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [term 1 LEADER]: Becoming Leader. State: Replica: 2e345d485c554efa8811e04f95f43141, State: Running, Role: LEADER
I20260620 08:14:12.643016 27831 consensus_queue.cc:237] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.643520 27848 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: TxnStatusTablet state changed. Reason: New leader 2e345d485c554efa8811e04f95f43141. Latest consensus state: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.643632 27848 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:12.643831 27850 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:14:12.643885 27317 catalog_manager.cc:5495] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e345d485c554efa8811e04f95f43141 (127.24.249.131). New cstate: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } health_report { overall_health: HEALTHY } } }
I20260620 08:14:12.643997 27850 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:14:12.664716 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.662177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.24.249.131:35687)
0620 08:14:12.662243 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.662245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.662287 (+ 42us) server_negotiation.cc:407] Connection header received
0620 08:14:12.662385 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.662387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.662406 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.662479 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:12.662601 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.663085 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.663852 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.664038 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.664114 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.664350 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:12.664460 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:12.664553 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.664597 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.664618 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:14:12.664716 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.662108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35687 (local address 127.0.0.1:58266)
0620 08:14:12.662237 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.662249 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.662309 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.662484 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.662485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.662486 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:12.662540 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.662543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.663166 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.663168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.663758 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.663759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.664118 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.664119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.664195 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.664217 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:12.664528 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:12.664539 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.664585 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.664640 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 08:14:12.666725 27476 raft_consensus.cc:1261] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Refusing update from remote peer 2e345d485c554efa8811e04f95f43141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:12.666749 27616 raft_consensus.cc:1261] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Refusing update from remote peer 2e345d485c554efa8811e04f95f43141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:12.667060 27848 consensus_queue.cc:1035] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:12.667201 27831 consensus_queue.cc:1035] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:12.668382 27833 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: TxnStatusTablet state changed. Reason: New leader 2e345d485c554efa8811e04f95f43141. Latest consensus state: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.668442 27832 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: TxnStatusTablet state changed. Reason: New leader 2e345d485c554efa8811e04f95f43141. Latest consensus state: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.668498 27833 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:12.668526 27832 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:12.669839 27848 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.669965 27848 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:12.670035 27853 mvcc.cc:205] Tried to move back new op lower bound from 7298839562920075264 to 7298839562826391552. Current Snapshot: MvccSnapshot[applied={T|T < 7298839562920075264}]
I20260620 08:14:12.670107 27832 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.670192 27832 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P b247f2f5566c4eb19503a9bfa694647c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:12.670583 27831 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.670667 27831 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P 2e345d485c554efa8811e04f95f43141: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:12.671080 27833 tablet_replica.cc:439] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e345d485c554efa8811e04f95f43141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } } }
I20260620 08:14:12.671165 27833 tablet_replica.cc:442] T d50cecb4f56642809e1accc73059a4ec P 8097dd42a4f54fb6b3256f292807d5ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:12.672253 27854 mvcc.cc:205] Tried to move back new op lower bound from 7298839562920075264 to 7298839562826391552. Current Snapshot: MvccSnapshot[applied={T|T < 7298839562920075264}]
I20260620 08:14:12.676378 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.673728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.24.249.129:39479)
0620 08:14:12.673918 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.673920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.673932 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:12.673972 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.673972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.673988 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.674079 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:12.674198 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.674691 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.675548 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.675777 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.675853 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.676022 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:12.676137 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:12.676229 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.676264 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.676288 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 08:14:12.681921 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.679089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:37563 (local address 127.24.249.131:35687)
0620 08:14:12.679165 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.679166 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.679229 (+ 63us) server_negotiation.cc:407] Connection header received
0620 08:14:12.679314 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.679314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.679323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.679359 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:12.679496 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.679981 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.681145 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.681363 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.681433 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.681641 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:12.681668 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:12.681722 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.681787 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.681806 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:14:12.681949 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.678969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35687 (local address 127.24.249.129:37563)
0620 08:14:12.679163 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.679174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.679245 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.679371 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.679372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.679374 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:12.679438 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.679440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.680053 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.680053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.681039 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.681042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.681442 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.681443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.681518 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.681535 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:12.681728 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:12.681731 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.681779 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.681825 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
I20260620 08:14:12.684542 27616 raft_consensus.cc:1261] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Refusing update from remote peer 8097dd42a4f54fb6b3256f292807d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:12.684835 27836 consensus_queue.cc:1035] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:12.684978 27756 raft_consensus.cc:1261] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Refusing update from remote peer 8097dd42a4f54fb6b3256f292807d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:12.685197 27836 consensus_queue.cc:1035] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:12.687052 27855 mvcc.cc:205] Tried to move back new op lower bound from 7298839562993786880 to 7298839562164768768. Current Snapshot: MvccSnapshot[applied={T|T < 7298839562993786880}]
I20260620 08:14:12.699931 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.697028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:55415 (local address 127.24.249.129:39479)
0620 08:14:12.697447 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:12.697450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:12.697457 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:12.697481 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:12.697482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:12.697492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:12.697568 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:12.697810 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.698344 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.699187 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.699398 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.699478 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.699659 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:12.699697 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:12.699744 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:12.699822 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 08:14:12.699847 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392}
I20260620 08:14:12.699934 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:12.696997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39479 (local address 127.24.249.131:55415)
0620 08:14:12.697100 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:12.697112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:12.697154 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:12.697574 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:12.697575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:12.697576 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:12.697714 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.697717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.698428 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.698429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.699042 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:12.699044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:12.699483 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:12.699484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:12.699552 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:12.699575 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:12.699769 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:12.699772 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:12.699817 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 08:14:12.699866 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 08:14:12.709581 27317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:12.709735 27317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:12.710814 27317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Sending DeleteTablet for 3 replicas of tablet c18f7d69417348ddb831025dde914905
I20260620 08:14:12.711228 27736 tablet_service.cc:1514] Processing DeleteTablet for tablet c18f7d69417348ddb831025dde914905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:12 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:58252
I20260620 08:14:12.711827 27456 tablet_service.cc:1514] Processing DeleteTablet for tablet c18f7d69417348ddb831025dde914905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:12 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:56462
I20260620 08:14:12.712154 27317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:14:12.712163 27596 tablet_service.cc:1514] Processing DeleteTablet for tablet c18f7d69417348ddb831025dde914905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:12 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:60976
W20260620 08:14:12.712311 27317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:12.712754 27873 tablet_replica.cc:330] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: stopping tablet replica
I20260620 08:14:12.712873 27873 raft_consensus.cc:2229] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:12.712986 27873 raft_consensus.cc:2258] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:12.713230 27872 tablet_replica.cc:330] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: stopping tablet replica
I20260620 08:14:12.713340 27872 raft_consensus.cc:2229] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:12.713392 27872 raft_consensus.cc:2258] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:12.713833 27736 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2d1c93a18b47af8cbc5df783b52be8 (DEFAULT_TABLE table=test-table [id=c9c352c768bc44faaef708b3c8296828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:12.713994 27736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2d1c93a18b47af8cbc5df783b52be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.715474 27456 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2d1c93a18b47af8cbc5df783b52be8 (DEFAULT_TABLE table=test-table [id=c9c352c768bc44faaef708b3c8296828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:12.715595 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2d1c93a18b47af8cbc5df783b52be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.716941 27826 tablet_bootstrap.cc:492] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: Bootstrap starting.
I20260620 08:14:12.717548 27826 tablet_bootstrap.cc:654] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.717965 27827 tablet_bootstrap.cc:492] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: Bootstrap starting.
I20260620 08:14:12.718406 27827 tablet_bootstrap.cc:654] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.713786 27596 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2d1c93a18b47af8cbc5df783b52be8 (DEFAULT_TABLE table=test-table [id=c9c352c768bc44faaef708b3c8296828]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:12.719080 27596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2d1c93a18b47af8cbc5df783b52be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:12.719128 27827 tablet_bootstrap.cc:492] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: No bootstrap required, opened a new log
I20260620 08:14:12.719175 27827 ts_tablet_manager.cc:1397] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:12.719321 27827 raft_consensus.cc:348] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.719394 27827 raft_consensus.cc:374] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.719414 27827 raft_consensus.cc:729] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8097dd42a4f54fb6b3256f292807d5ab, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.719458 27827 consensus_queue.cc:260] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.719550 27827 ts_tablet_manager.cc:1428] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:12.720510 27825 tablet_bootstrap.cc:492] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: Bootstrap starting.
I20260620 08:14:12.713466 27874 tablet_replica.cc:330] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: stopping tablet replica
I20260620 08:14:12.720739 27874 raft_consensus.cc:2229] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:12.720809 27874 raft_consensus.cc:2258] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:12.720966 27825 tablet_bootstrap.cc:654] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:12.721428 27872 ts_tablet_manager.cc:1905] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:12.721616 27825 tablet_bootstrap.cc:492] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: No bootstrap required, opened a new log
I20260620 08:14:12.721681 27825 ts_tablet_manager.cc:1397] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:12.721845 27825 raft_consensus.cc:348] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.721966 27825 raft_consensus.cc:374] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.721995 27825 raft_consensus.cc:729] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b247f2f5566c4eb19503a9bfa694647c, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.722038 27825 consensus_queue.cc:260] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.722129 27825 ts_tablet_manager.cc:1428] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:12.722366 27874 ts_tablet_manager.cc:1905] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:12.723146 27872 ts_tablet_manager.cc:1918] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:12.723228 27872 log.cc:1196] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/c18f7d69417348ddb831025dde914905
I20260620 08:14:12.723544 27872 ts_tablet_manager.cc:1939] T c18f7d69417348ddb831025dde914905 P 2e345d485c554efa8811e04f95f43141: Deleting consensus metadata
I20260620 08:14:12.723978 27304 catalog_manager.cc:4841] TS 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687): tablet c18f7d69417348ddb831025dde914905 (table test-table [id=d8a1e1ce722f44dbab6997e4d40bc1c9]) successfully deleted
I20260620 08:14:12.724620 27874 ts_tablet_manager.cc:1918] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:12.724711 27874 log.cc:1196] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/c18f7d69417348ddb831025dde914905
I20260620 08:14:12.724985 27874 ts_tablet_manager.cc:1939] T c18f7d69417348ddb831025dde914905 P b247f2f5566c4eb19503a9bfa694647c: Deleting consensus metadata
I20260620 08:14:12.725435 27304 catalog_manager.cc:4841] TS b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655): tablet c18f7d69417348ddb831025dde914905 (table test-table [id=d8a1e1ce722f44dbab6997e4d40bc1c9]) successfully deleted
I20260620 08:14:12.725708 27826 tablet_bootstrap.cc:492] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: No bootstrap required, opened a new log
I20260620 08:14:12.725754 27826 ts_tablet_manager.cc:1397] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260620 08:14:12.725903 27826 raft_consensus.cc:348] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.725980 27826 raft_consensus.cc:374] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:12.726002 27826 raft_consensus.cc:729] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e345d485c554efa8811e04f95f43141, State: Initialized, Role: FOLLOWER
I20260620 08:14:12.726043 27826 consensus_queue.cc:260] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.726146 27826 ts_tablet_manager.cc:1428] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:12.733971 27873 ts_tablet_manager.cc:1905] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:12.736009 27873 ts_tablet_manager.cc:1918] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:12.736269 27873 log.cc:1196] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/c18f7d69417348ddb831025dde914905
I20260620 08:14:12.736550 27873 ts_tablet_manager.cc:1939] T c18f7d69417348ddb831025dde914905 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting consensus metadata
I20260620 08:14:12.737002 27303 catalog_manager.cc:4841] TS 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479): tablet c18f7d69417348ddb831025dde914905 (table test-table [id=d8a1e1ce722f44dbab6997e4d40bc1c9]) successfully deleted
I20260620 08:14:12.778443 27868 raft_consensus.cc:482] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:12.778550 27868 raft_consensus.cc:504] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.778697 27868 leader_election.cc:288] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:12.778941 27616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d1c93a18b47af8cbc5df783b52be8" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b247f2f5566c4eb19503a9bfa694647c" is_pre_election: true
I20260620 08:14:12.779032 27616 raft_consensus.cc:2454] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 0.
I20260620 08:14:12.779027 27756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d1c93a18b47af8cbc5df783b52be8" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141" is_pre_election: true
I20260620 08:14:12.779140 27756 raft_consensus.cc:2454] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 0.
I20260620 08:14:12.779229 27408 leader_election.cc:302] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8097dd42a4f54fb6b3256f292807d5ab, b247f2f5566c4eb19503a9bfa694647c; no voters:
I20260620 08:14:12.779347 27868 raft_consensus.cc:2781] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:12.779383 27868 raft_consensus.cc:482] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:12.779402 27868 raft_consensus.cc:3037] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.780272 27868 raft_consensus.cc:504] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.780408 27868 leader_election.cc:288] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 election: Requested vote from peers b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:12.780627 27616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d1c93a18b47af8cbc5df783b52be8" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b247f2f5566c4eb19503a9bfa694647c"
I20260620 08:14:12.780644 27756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d1c93a18b47af8cbc5df783b52be8" candidate_uuid: "8097dd42a4f54fb6b3256f292807d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141"
I20260620 08:14:12.780712 27616 raft_consensus.cc:3037] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.780722 27756 raft_consensus.cc:3037] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:12.781487 27756 raft_consensus.cc:2454] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 1.
I20260620 08:14:12.781546 27616 raft_consensus.cc:2454] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8097dd42a4f54fb6b3256f292807d5ab in term 1.
I20260620 08:14:12.781647 27408 leader_election.cc:302] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e345d485c554efa8811e04f95f43141, 8097dd42a4f54fb6b3256f292807d5ab; no voters:
I20260620 08:14:12.781764 27868 raft_consensus.cc:2781] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:12.781817 27868 raft_consensus.cc:686] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 LEADER]: Becoming Leader. State: Replica: 8097dd42a4f54fb6b3256f292807d5ab, State: Running, Role: LEADER
I20260620 08:14:12.781884 27868 consensus_queue.cc:237] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:12.782462 27317 catalog_manager.cc:5495] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129). New cstate: current_term: 1 leader_uuid: "8097dd42a4f54fb6b3256f292807d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:12.794122 27756 raft_consensus.cc:1261] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Refusing update from remote peer 8097dd42a4f54fb6b3256f292807d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:12.794425 27868 consensus_queue.cc:1035] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:12.796268 27616 raft_consensus.cc:1261] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Refusing update from remote peer 8097dd42a4f54fb6b3256f292807d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:12.796473 27836 consensus_queue.cc:1035] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:14:12.800458 27855 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 08:14:14.321096 27317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:14.321218 27317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:14.322048 27317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Sending DeleteTablet for 3 replicas of tablet 5e2d1c93a18b47af8cbc5df783b52be8
I20260620 08:14:14.322330 27596 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2d1c93a18b47af8cbc5df783b52be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:14 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:60976
I20260620 08:14:14.322329 27456 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2d1c93a18b47af8cbc5df783b52be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:14 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:56462
I20260620 08:14:14.322371 27736 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2d1c93a18b47af8cbc5df783b52be8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:14 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:58252
I20260620 08:14:14.322539 27905 tablet_replica.cc:330] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: stopping tablet replica
I20260620 08:14:14.322590 27905 raft_consensus.cc:2229] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:14.322630 27905 raft_consensus.cc:2258] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:14.322733 27904 tablet_replica.cc:330] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: stopping tablet replica
I20260620 08:14:14.322784 27904 raft_consensus.cc:2229] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:14.322842 27904 raft_consensus.cc:2258] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:14.322939 27906 tablet_replica.cc:330] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: stopping tablet replica
I20260620 08:14:14.322952 27905 ts_tablet_manager.cc:1905] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:14.322991 27906 raft_consensus.cc:2229] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:14.323027 27906 raft_consensus.cc:2258] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:14.323218 27904 ts_tablet_manager.cc:1905] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:14.325104 27904 ts_tablet_manager.cc:1918] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:14:14.325155 27904 log.cc:1196] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/5e2d1c93a18b47af8cbc5df783b52be8
I20260620 08:14:14.325443 27904 ts_tablet_manager.cc:1939] T 5e2d1c93a18b47af8cbc5df783b52be8 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting consensus metadata
I20260620 08:14:14.325760 27303 catalog_manager.cc:4841] TS 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479): tablet 5e2d1c93a18b47af8cbc5df783b52be8 (table test-table [id=c9c352c768bc44faaef708b3c8296828]) successfully deleted
I20260620 08:14:14.325798 27906 ts_tablet_manager.cc:1905] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:14.326242 27905 ts_tablet_manager.cc:1918] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:14:14.326296 27905 log.cc:1196] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/5e2d1c93a18b47af8cbc5df783b52be8
I20260620 08:14:14.326462 27905 ts_tablet_manager.cc:1939] T 5e2d1c93a18b47af8cbc5df783b52be8 P b247f2f5566c4eb19503a9bfa694647c: Deleting consensus metadata
I20260620 08:14:14.327232 27304 catalog_manager.cc:4841] TS b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655): tablet 5e2d1c93a18b47af8cbc5df783b52be8 (table test-table [id=c9c352c768bc44faaef708b3c8296828]) successfully deleted
I20260620 08:14:14.327298 27906 ts_tablet_manager.cc:1918] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:14:14.327404 27906 log.cc:1196] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/5e2d1c93a18b47af8cbc5df783b52be8
I20260620 08:14:14.327623 27906 ts_tablet_manager.cc:1939] T 5e2d1c93a18b47af8cbc5df783b52be8 P 2e345d485c554efa8811e04f95f43141: Deleting consensus metadata
I20260620 08:14:14.328034 27304 catalog_manager.cc:4841] TS 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687): tablet 5e2d1c93a18b47af8cbc5df783b52be8 (table test-table [id=c9c352c768bc44faaef708b3c8296828]) successfully deleted
Jun 20 08:14:14 dist-test-slave-65w8 krb5kdc[27274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:14:14 dist-test-slave-65w8 krb5kdc[27274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943254, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:14.356348 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.342378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.24.249.190:39631)
0620 08:14:14.342583 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:14.342586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:14.342603 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:14:14.342695 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:14.342697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:14.342712 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:14.342799 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:14.343720 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.344250 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.345295 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.345502 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.345569 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.353565 (+ 7996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:14.353567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:14.353567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:14.353576 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:14.355162 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:14.355517 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:14.355518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:14.355519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:14.355573 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:14.355788 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:14.355789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:14.355790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:14.355928 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:14.356023 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:14.356175 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 08:14:14.356254 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260620 08:14:14.358027 27317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:14.358217 27317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:14.359717 27456 tablet_service.cc:1467] Processing CreateTablet for tablet 75e7d9be515c427b89c9f6f1defe8255 (DEFAULT_TABLE table=test-table [id=80855bcfa1b1444584a71f09b72af046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:14.359784 27596 tablet_service.cc:1467] Processing CreateTablet for tablet 75e7d9be515c427b89c9f6f1defe8255 (DEFAULT_TABLE table=test-table [id=80855bcfa1b1444584a71f09b72af046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:14.359786 27736 tablet_service.cc:1467] Processing CreateTablet for tablet 75e7d9be515c427b89c9f6f1defe8255 (DEFAULT_TABLE table=test-table [id=80855bcfa1b1444584a71f09b72af046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:14.359892 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e7d9be515c427b89c9f6f1defe8255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:14.359915 27596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e7d9be515c427b89c9f6f1defe8255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:14.359985 27736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e7d9be515c427b89c9f6f1defe8255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:14.361418 27916 tablet_bootstrap.cc:492] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: Bootstrap starting.
I20260620 08:14:14.361474 27915 tablet_bootstrap.cc:492] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: Bootstrap starting.
I20260620 08:14:14.361624 27917 tablet_bootstrap.cc:492] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: Bootstrap starting.
I20260620 08:14:14.361754 27916 tablet_bootstrap.cc:654] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:14.362013 27915 tablet_bootstrap.cc:654] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:14.362129 27917 tablet_bootstrap.cc:654] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:14.362458 27916 tablet_bootstrap.cc:492] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: No bootstrap required, opened a new log
I20260620 08:14:14.362501 27916 ts_tablet_manager.cc:1397] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:14.362619 27916 raft_consensus.cc:348] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.362707 27916 raft_consensus.cc:374] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:14.362730 27916 raft_consensus.cc:729] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e345d485c554efa8811e04f95f43141, State: Initialized, Role: FOLLOWER
I20260620 08:14:14.362749 27917 tablet_bootstrap.cc:492] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: No bootstrap required, opened a new log
I20260620 08:14:14.362777 27916 consensus_queue.cc:260] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.362823 27917 ts_tablet_manager.cc:1397] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:14.362881 27916 ts_tablet_manager.cc:1428] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:14.362950 27917 raft_consensus.cc:348] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.363023 27917 raft_consensus.cc:374] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:14.363049 27917 raft_consensus.cc:729] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8097dd42a4f54fb6b3256f292807d5ab, State: Initialized, Role: FOLLOWER
I20260620 08:14:14.363124 27917 consensus_queue.cc:260] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.363229 27917 ts_tablet_manager.cc:1428] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:14.363462 27915 tablet_bootstrap.cc:492] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: No bootstrap required, opened a new log
I20260620 08:14:14.363528 27915 ts_tablet_manager.cc:1397] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:14.363744 27915 raft_consensus.cc:348] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.363853 27915 raft_consensus.cc:374] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:14.363878 27915 raft_consensus.cc:729] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b247f2f5566c4eb19503a9bfa694647c, State: Initialized, Role: FOLLOWER
I20260620 08:14:14.363932 27915 consensus_queue.cc:260] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.364130 27915 ts_tablet_manager.cc:1428] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:14.513576 27921 raft_consensus.cc:482] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:14.513724 27921 raft_consensus.cc:504] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.514169 27921 leader_election.cc:288] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:14.517491 27923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.514347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39479 (local address 127.24.249.130:45871)
0620 08:14:14.514588 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:14.514600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:14.514676 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:14.514842 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:14.514843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:14.514845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:14.514924 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.514931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.515622 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.515623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.516586 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.516589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.517201 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.517202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.517283 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.517288 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:14.517348 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:14:14.517389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":147,"threads_started":1}
I20260620 08:14:14.517480 27924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.514452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:45871 (local address 127.24.249.129:39479)
0620 08:14:14.514613 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:14.514627 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:14.514656 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:14:14.514760 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:14.514761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:14.514773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:14.514839 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:14.514995 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.515516 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.516693 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.517111 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.517202 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.517211 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:14.517356 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 08:14:14.517404 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260620 08:14:14.517741 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7d9be515c427b89c9f6f1defe8255" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097dd42a4f54fb6b3256f292807d5ab" is_pre_election: true
I20260620 08:14:14.517891 27476 raft_consensus.cc:2454] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 0.
I20260620 08:14:14.517951 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.514421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35687 (local address 127.24.249.130:45331)
0620 08:14:14.514607 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:14.514619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:14.514686 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:14.514889 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:14.514890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:14.514892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:14.515302 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.515305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.516048 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.516049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.516841 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.516843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.517653 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.517654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.517730 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.517734 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:14.517802 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:14:14.517841 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260620 08:14:14.518126 27547 leader_election.cc:302] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8097dd42a4f54fb6b3256f292807d5ab, b247f2f5566c4eb19503a9bfa694647c; no voters:
I20260620 08:14:14.518172 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.514467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:45331 (local address 127.24.249.131:35687)
0620 08:14:14.514631 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:14.514634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:14.514668 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:14:14.514799 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:14.514800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:14.514809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:14.514888 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:14.515388 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.515966 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.516933 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.517506 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.517602 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.517613 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:14.518030 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 08:14:14.518083 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260620 08:14:14.518287 27921 raft_consensus.cc:2781] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:14.518361 27921 raft_consensus.cc:482] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:14.518384 27921 raft_consensus.cc:3037] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:14.519062 27921 raft_consensus.cc:504] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.519209 27921 leader_election.cc:288] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 election: Requested vote from peers 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:14.519300 27756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7d9be515c427b89c9f6f1defe8255" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141" is_pre_election: true
I20260620 08:14:14.519377 27756 raft_consensus.cc:2454] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 0.
I20260620 08:14:14.519366 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7d9be515c427b89c9f6f1defe8255" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097dd42a4f54fb6b3256f292807d5ab"
I20260620 08:14:14.519393 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7d9be515c427b89c9f6f1defe8255" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141"
I20260620 08:14:14.519439 27476 raft_consensus.cc:3037] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:14.519448 27755 raft_consensus.cc:3037] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:14.520092 27755 raft_consensus.cc:2454] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 1.
I20260620 08:14:14.520102 27476 raft_consensus.cc:2454] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 1.
I20260620 08:14:14.520303 27548 leader_election.cc:302] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e345d485c554efa8811e04f95f43141, b247f2f5566c4eb19503a9bfa694647c; no voters:
I20260620 08:14:14.520409 27921 raft_consensus.cc:2781] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:14.520573 27921 raft_consensus.cc:686] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 1 LEADER]: Becoming Leader. State: Replica: b247f2f5566c4eb19503a9bfa694647c, State: Running, Role: LEADER
I20260620 08:14:14.520654 27921 consensus_queue.cc:237] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.521492 27319 catalog_manager.cc:5495] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c reported cstate change: term changed from 0 to 1, leader changed from <none> to b247f2f5566c4eb19503a9bfa694647c (127.24.249.130). New cstate: current_term: 1 leader_uuid: "b247f2f5566c4eb19503a9bfa694647c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:14.548273 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.545247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.24.249.130:40655)
0620 08:14:14.545453 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:14.545455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:14.545464 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:14.545510 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:14.545511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:14.545531 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:14.545607 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:14.545828 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.546439 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.547287 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.547540 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.547674 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.547862 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:14.547997 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:14.548082 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:14.548149 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:14:14.548171 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":154,"threads_started":1}
I20260620 08:14:14.553205 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.550830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:53151 (local address 127.24.249.131:35687)
0620 08:14:14.550891 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:14.550893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:14.551000 (+ 107us) server_negotiation.cc:407] Connection header received
0620 08:14:14.551074 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:14.551075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:14.551085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:14.551152 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:14.551273 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.551754 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.552500 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.552708 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.552782 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.552953 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:14.552992 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:14.553048 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:14.553105 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 08:14:14.553125 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:14:14.553254 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.550731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35687 (local address 127.24.249.130:53151)
0620 08:14:14.550938 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:14.550951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:14.551015 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:14.551155 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:14.551156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:14.551157 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:14.551214 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.551215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.551835 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.551836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.552421 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.552422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.552782 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.552783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.552847 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.552865 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:14.553050 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:14.553052 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:14.553099 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 08:14:14.553142 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":108,"threads_started":1}
I20260620 08:14:14.555549 27755 raft_consensus.cc:1261] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Refusing update from remote peer b247f2f5566c4eb19503a9bfa694647c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:14.555544 27476 raft_consensus.cc:1261] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Refusing update from remote peer b247f2f5566c4eb19503a9bfa694647c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:14.555830 27921 consensus_queue.cc:1035] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:14.555991 27926 consensus_queue.cc:1035] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:14.563763 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.561225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:32941 (local address 127.24.249.130:40655)
0620 08:14:14.561265 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:14.561267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:14.561372 (+ 105us) server_negotiation.cc:407] Connection header received
0620 08:14:14.561445 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:14.561446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:14.561455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:14.561506 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:14.561616 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.562154 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.563016 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.563223 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.563303 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.563508 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:14.563543 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:14.563607 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:14.563666 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 08:14:14.563689 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:14.563788 27937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:14.561156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:40655 (local address 127.24.249.131:32941)
0620 08:14:14.561305 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:14.561316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:14.561387 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:14.561509 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:14.561510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:14.561511 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:14.561559 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.561560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.562235 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.562236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.562916 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:14.562918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:14.563357 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:14.563358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:14.563416 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:14.563421 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:14.563606 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:14.563608 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:14.563662 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 08:14:14.563713 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":104,"threads_started":1}
I20260620 08:14:14.570091 27319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:14.570211 27319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:14.570894 27319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Sending DeleteTablet for 3 replicas of tablet 75e7d9be515c427b89c9f6f1defe8255
I20260620 08:14:14.571146 27456 tablet_service.cc:1514] Processing DeleteTablet for tablet 75e7d9be515c427b89c9f6f1defe8255 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:14 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:56462
I20260620 08:14:14.571163 27736 tablet_service.cc:1514] Processing DeleteTablet for tablet 75e7d9be515c427b89c9f6f1defe8255 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:14 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:58252
I20260620 08:14:14.571213 27596 tablet_service.cc:1514] Processing DeleteTablet for tablet 75e7d9be515c427b89c9f6f1defe8255 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:14 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:60976
I20260620 08:14:14.571251 27904 tablet_replica.cc:330] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: stopping tablet replica
I20260620 08:14:14.571262 27906 tablet_replica.cc:330] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: stopping tablet replica
I20260620 08:14:14.571313 27904 raft_consensus.cc:2229] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:14.571317 27906 raft_consensus.cc:2229] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:14.571323 27905 tablet_replica.cc:330] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: stopping tablet replica
I20260620 08:14:14.571349 27904 raft_consensus.cc:2258] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:14.571354 27906 raft_consensus.cc:2258] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:14.571373 27905 raft_consensus.cc:2229] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:14.571458 27905 raft_consensus.cc:2258] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:14.571755 27906 ts_tablet_manager.cc:1905] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:14.571766 27904 ts_tablet_manager.cc:1905] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:14.571780 27905 ts_tablet_manager.cc:1905] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:14.573130 27905 ts_tablet_manager.cc:1918] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:14.573205 27905 log.cc:1196] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/75e7d9be515c427b89c9f6f1defe8255
I20260620 08:14:14.573346 27906 ts_tablet_manager.cc:1918] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:14.573369 27904 ts_tablet_manager.cc:1918] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:14.573413 27906 log.cc:1196] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/75e7d9be515c427b89c9f6f1defe8255
I20260620 08:14:14.573413 27904 log.cc:1196] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/75e7d9be515c427b89c9f6f1defe8255
I20260620 08:14:14.573454 27905 ts_tablet_manager.cc:1939] T 75e7d9be515c427b89c9f6f1defe8255 P b247f2f5566c4eb19503a9bfa694647c: Deleting consensus metadata
I20260620 08:14:14.573642 27904 ts_tablet_manager.cc:1939] T 75e7d9be515c427b89c9f6f1defe8255 P 8097dd42a4f54fb6b3256f292807d5ab: Deleting consensus metadata
I20260620 08:14:14.573642 27906 ts_tablet_manager.cc:1939] T 75e7d9be515c427b89c9f6f1defe8255 P 2e345d485c554efa8811e04f95f43141: Deleting consensus metadata
I20260620 08:14:14.573765 27304 catalog_manager.cc:4841] TS b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655): tablet 75e7d9be515c427b89c9f6f1defe8255 (table test-table [id=80855bcfa1b1444584a71f09b72af046]) successfully deleted
I20260620 08:14:14.573741 27319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:14.573916 27319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:14.574019 27303 catalog_manager.cc:4841] TS 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479): tablet 75e7d9be515c427b89c9f6f1defe8255 (table test-table [id=80855bcfa1b1444584a71f09b72af046]) successfully deleted
I20260620 08:14:14.574404 27304 catalog_manager.cc:4841] TS 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687): tablet 75e7d9be515c427b89c9f6f1defe8255 (table test-table [id=80855bcfa1b1444584a71f09b72af046]) successfully deleted
I20260620 08:14:14.575150 27456 tablet_service.cc:1467] Processing CreateTablet for tablet a05acab693194ed59274f506304d7826 (DEFAULT_TABLE table=test-table [id=4ca3453d434a4ac79b469169fd42f7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:14.575165 27736 tablet_service.cc:1467] Processing CreateTablet for tablet a05acab693194ed59274f506304d7826 (DEFAULT_TABLE table=test-table [id=4ca3453d434a4ac79b469169fd42f7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:14.575174 27596 tablet_service.cc:1467] Processing CreateTablet for tablet a05acab693194ed59274f506304d7826 (DEFAULT_TABLE table=test-table [id=4ca3453d434a4ac79b469169fd42f7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:14.575264 27736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05acab693194ed59274f506304d7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:14.575264 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05acab693194ed59274f506304d7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:14.575310 27596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05acab693194ed59274f506304d7826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:14.576100 27915 tablet_bootstrap.cc:492] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: Bootstrap starting.
I20260620 08:14:14.576421 27916 tablet_bootstrap.cc:492] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: Bootstrap starting.
I20260620 08:14:14.576538 27915 tablet_bootstrap.cc:654] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:14.576783 27916 tablet_bootstrap.cc:654] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:14.577044 27917 tablet_bootstrap.cc:492] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab: Bootstrap starting.
I20260620 08:14:14.577569 27917 tablet_bootstrap.cc:654] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:14.577741 27916 tablet_bootstrap.cc:492] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: No bootstrap required, opened a new log
I20260620 08:14:14.577786 27916 ts_tablet_manager.cc:1397] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:14.577931 27916 raft_consensus.cc:348] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.578006 27916 raft_consensus.cc:374] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:14.578029 27916 raft_consensus.cc:729] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e345d485c554efa8811e04f95f43141, State: Initialized, Role: FOLLOWER
I20260620 08:14:14.578044 27915 tablet_bootstrap.cc:492] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: No bootstrap required, opened a new log
I20260620 08:14:14.578083 27915 ts_tablet_manager.cc:1397] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:14.578068 27916 consensus_queue.cc:260] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.578161 27916 ts_tablet_manager.cc:1428] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:14.578269 27915 raft_consensus.cc:348] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.578330 27915 raft_consensus.cc:374] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:14.578356 27915 raft_consensus.cc:729] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b247f2f5566c4eb19503a9bfa694647c, State: Initialized, Role: FOLLOWER
I20260620 08:14:14.578402 27915 consensus_queue.cc:260] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.578512 27915 ts_tablet_manager.cc:1428] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:14.578711 27917 tablet_bootstrap.cc:492] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab: No bootstrap required, opened a new log
I20260620 08:14:14.578749 27917 ts_tablet_manager.cc:1397] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:14.578890 27917 raft_consensus.cc:348] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.578956 27917 raft_consensus.cc:374] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:14.578980 27917 raft_consensus.cc:729] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8097dd42a4f54fb6b3256f292807d5ab, State: Initialized, Role: FOLLOWER
I20260620 08:14:14.579033 27917 consensus_queue.cc:260] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.579129 27917 ts_tablet_manager.cc:1428] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:14.783627 27921 raft_consensus.cc:482] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:14.783740 27921 raft_consensus.cc:504] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.783914 27921 leader_election.cc:288] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:14.784180 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05acab693194ed59274f506304d7826" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097dd42a4f54fb6b3256f292807d5ab" is_pre_election: true
I20260620 08:14:14.784168 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05acab693194ed59274f506304d7826" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141" is_pre_election: true
I20260620 08:14:14.784282 27476 raft_consensus.cc:2454] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 0.
I20260620 08:14:14.784292 27755 raft_consensus.cc:2454] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 0.
I20260620 08:14:14.784461 27548 leader_election.cc:302] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e345d485c554efa8811e04f95f43141, b247f2f5566c4eb19503a9bfa694647c; no voters:
I20260620 08:14:14.784583 27921 raft_consensus.cc:2781] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:14.784626 27921 raft_consensus.cc:482] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:14.784649 27921 raft_consensus.cc:3037] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:14.785354 27921 raft_consensus.cc:504] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.785492 27921 leader_election.cc:288] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 election: Requested vote from peers 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479), 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687)
I20260620 08:14:14.785646 27755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05acab693194ed59274f506304d7826" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e345d485c554efa8811e04f95f43141"
I20260620 08:14:14.785713 27755 raft_consensus.cc:3037] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:14.785662 27476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05acab693194ed59274f506304d7826" candidate_uuid: "b247f2f5566c4eb19503a9bfa694647c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097dd42a4f54fb6b3256f292807d5ab"
I20260620 08:14:14.785759 27476 raft_consensus.cc:3037] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:14.786372 27755 raft_consensus.cc:2454] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 1.
I20260620 08:14:14.786489 27476 raft_consensus.cc:2454] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b247f2f5566c4eb19503a9bfa694647c in term 1.
I20260620 08:14:14.786541 27548 leader_election.cc:302] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e345d485c554efa8811e04f95f43141, b247f2f5566c4eb19503a9bfa694647c; no voters:
I20260620 08:14:14.786717 27921 raft_consensus.cc:2781] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:14.786793 27921 raft_consensus.cc:686] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 1 LEADER]: Becoming Leader. State: Replica: b247f2f5566c4eb19503a9bfa694647c, State: Running, Role: LEADER
I20260620 08:14:14.786850 27921 consensus_queue.cc:237] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } }
I20260620 08:14:14.787461 27317 catalog_manager.cc:5495] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c reported cstate change: term changed from 0 to 1, leader changed from <none> to b247f2f5566c4eb19503a9bfa694647c (127.24.249.130). New cstate: current_term: 1 leader_uuid: "b247f2f5566c4eb19503a9bfa694647c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b247f2f5566c4eb19503a9bfa694647c" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:14.807478 27755 raft_consensus.cc:1261] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Refusing update from remote peer b247f2f5566c4eb19503a9bfa694647c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:14.807478 27476 raft_consensus.cc:1261] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab [term 1 FOLLOWER]: Refusing update from remote peer b247f2f5566c4eb19503a9bfa694647c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:14.807698 27921 consensus_queue.cc:1035] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8097dd42a4f54fb6b3256f292807d5ab" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:14.807803 27926 consensus_queue.cc:1035] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e345d485c554efa8811e04f95f43141" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:14:14.812151 27887 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 08:14:16.330179 27319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:16.330300 27319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:16.331187 27319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d9f6bd3180540c18cb6c467b4677c70: Sending DeleteTablet for 3 replicas of tablet a05acab693194ed59274f506304d7826
I20260620 08:14:16.331439 27456 tablet_service.cc:1514] Processing DeleteTablet for tablet a05acab693194ed59274f506304d7826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:16 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:56462
I20260620 08:14:16.331521 27736 tablet_service.cc:1514] Processing DeleteTablet for tablet a05acab693194ed59274f506304d7826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:16 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:58252
I20260620 08:14:16.331620 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 27391
I20260620 08:14:16.331640 27967 tablet_replica.cc:330] T a05acab693194ed59274f506304d7826 P 8097dd42a4f54fb6b3256f292807d5ab: stopping tablet replica
I20260620 08:14:16.332127 27596 tablet_service.cc:1514] Processing DeleteTablet for tablet a05acab693194ed59274f506304d7826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:16 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:60976
I20260620 08:14:16.332276 27968 tablet_replica.cc:330] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: stopping tablet replica
I20260620 08:14:16.332334 27968 raft_consensus.cc:2229] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:16.332350 27969 tablet_replica.cc:330] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: stopping tablet replica
I20260620 08:14:16.332376 27968 raft_consensus.cc:2258] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:16.332412 27969 raft_consensus.cc:2229] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:16.332470 27969 raft_consensus.cc:2258] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:16.332727 27969 ts_tablet_manager.cc:1905] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:16.332728 27968 ts_tablet_manager.cc:1905] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:16.334174 27969 ts_tablet_manager.cc:1918] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:14:16.334213 27969 log.cc:1196] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/a05acab693194ed59274f506304d7826
I20260620 08:14:16.334376 27969 ts_tablet_manager.cc:1939] T a05acab693194ed59274f506304d7826 P b247f2f5566c4eb19503a9bfa694647c: Deleting consensus metadata
I20260620 08:14:16.334722 27304 catalog_manager.cc:4841] TS b247f2f5566c4eb19503a9bfa694647c (127.24.249.130:40655): tablet a05acab693194ed59274f506304d7826 (table test-table [id=4ca3453d434a4ac79b469169fd42f7ed]) successfully deleted
I20260620 08:14:16.334878 27968 ts_tablet_manager.cc:1918] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 08:14:16.334936 27968 log.cc:1196] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/a05acab693194ed59274f506304d7826
I20260620 08:14:16.335176 27968 ts_tablet_manager.cc:1939] T a05acab693194ed59274f506304d7826 P 2e345d485c554efa8811e04f95f43141: Deleting consensus metadata
I20260620 08:14:16.335635 27304 catalog_manager.cc:4841] TS 2e345d485c554efa8811e04f95f43141 (127.24.249.131:35687): tablet a05acab693194ed59274f506304d7826 (table test-table [id=4ca3453d434a4ac79b469169fd42f7ed]) successfully deleted
W20260620 08:14:16.340512 27303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:39479 (error 108)
I20260620 08:14:16.340694 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 27531
I20260620 08:14:16.341908 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:16.341655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39479 (local address 127.0.0.1:42510)
0620 08:14:16.341789 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:16.341837 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:39479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
W20260620 08:14:16.342737 27303 catalog_manager.cc:4584] TS 8097dd42a4f54fb6b3256f292807d5ab (127.24.249.129:39479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a05acab693194ed59274f506304d7826: Network error: Client connection negotiation failed: client connection to 127.24.249.129:39479: connect: Connection refused (error 111)
I20260620 08:14:16.347244 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 27671
I20260620 08:14:16.353541 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 27287
2026-06-20T08:14:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5901 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:16 dist-test-slave-65w8 krb5kdc[27974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:16 dist-test-slave-65w8 krb5kdc[27974](info): set up 2 sockets
Jun 20 08:14:16 dist-test-slave-65w8 krb5kdc[27974](info): commencing operation
krb5kdc: starting...
W20260620 08:14:17.462131 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:17 dist-test-slave-65w8 krb5kdc[27974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:17.589502 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:37199
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:17.672369 27987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:17.672593 27987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:17.672616 27987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:17.673743 27987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:17.673782 27987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:17.673797 27987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:17.673810 27987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:17.673822 27987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:17.675480 27987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:37199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:17.675678 27987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:17.675850 27987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:17.680902 27993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:17.681030 27987 server_base.cc:1034] running on GCE node
W20260620 08:14:17.681100 27995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:17.680927 27992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:17.681401 27987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:17.681704 27987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:17.682871 27987 hybrid_clock.cc:648] HybridClock initialized: now 1781943257682862 us; error 35 us; skew 500 ppm
Jun 20 08:14:17 dist-test-slave-65w8 krb5kdc[27974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943257, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:17.685508 27987 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:17.686069 27987 webserver.cc:460] Webserver started at http://127.24.249.190:43315/ using document root <none> and password file <none>
I20260620 08:14:17.686304 27987 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:17.686372 27987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:17.686530 27987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:17.687657 27987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45"
format_stamp: "Formatted at 2026-06-20 08:14:17 on dist-test-slave-65w8"
server_key: "0892244be24b9558f8a1489b96b82768"
server_key_iv: "07b536651316636100a761d2fd2c4844"
server_key_version: "encryptionkey@0"
I20260620 08:14:17.688022 27987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45"
format_stamp: "Formatted at 2026-06-20 08:14:17 on dist-test-slave-65w8"
server_key: "0892244be24b9558f8a1489b96b82768"
server_key_iv: "07b536651316636100a761d2fd2c4844"
server_key_version: "encryptionkey@0"
I20260620 08:14:17.689484 27987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:17.690505 28002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:17.690778 27987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:17.690907 27987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45"
format_stamp: "Formatted at 2026-06-20 08:14:17 on dist-test-slave-65w8"
server_key: "0892244be24b9558f8a1489b96b82768"
server_key_iv: "07b536651316636100a761d2fd2c4844"
server_key_version: "encryptionkey@0"
I20260620 08:14:17.690994 27987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:17.706919 27987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:17.708927 27987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:17.709101 27987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:17.713439 27987 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:37199
I20260620 08:14:17.713476 28054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:37199 every 8 connection(s)
I20260620 08:14:17.713847 27987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:17.714691 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 27987
I20260620 08:14:17.714787 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:17.714876 28055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:17.714984 25574 external_mini_cluster.cc:1428] Setting key 22b80e61c861bf72d28b62b1bc920d42
I20260620 08:14:17.718335 28055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45: Bootstrap starting.
I20260620 08:14:17.718978 28055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:17.719278 28055 log.cc:824] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:17.719974 28055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45: No bootstrap required, opened a new log
Jun 20 08:14:17 dist-test-slave-65w8 krb5kdc[27974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:17.722360 28055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37199 } }
I20260620 08:14:17.722514 28055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:17.722537 28055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907cf1e7747f4ffe9d6d17b2f7da3e45, State: Initialized, Role: FOLLOWER
I20260620 08:14:17.722694 28055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37199 } }
I20260620 08:14:17.722760 28055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:17.722806 28055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:17.722862 28055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:17.723634 28055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37199 } }
I20260620 08:14:17.723757 28055 leader_election.cc:302] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907cf1e7747f4ffe9d6d17b2f7da3e45; no voters:
I20260620 08:14:17.723953 28055 leader_election.cc:288] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:17.724067 28060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:17.724221 28060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [term 1 LEADER]: Becoming Leader. State: Replica: 907cf1e7747f4ffe9d6d17b2f7da3e45, State: Running, Role: LEADER
I20260620 08:14:17.724342 28055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:17.724495 28060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37199 } }
I20260620 08:14:17.724960 28061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907cf1e7747f4ffe9d6d17b2f7da3e45. Latest consensus state: current_term: 1 leader_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37199 } } }
I20260620 08:14:17.725129 28061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:17.726511 28075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:17.726573 28075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:17.726687 28069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:17.726897 28069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:17.727418 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:17.715532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.24.249.190:37199)
0620 08:14:17.715700 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:17.715710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:17.715745 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:14:17.716349 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:17.716356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:17.716404 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:17.716520 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:17.717386 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:17.718454 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:17.719304 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:17.719786 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:17.719908 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:17.722864 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:17.722868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:17.722868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:17.722882 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:17.725318 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:17.725770 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:17.725773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:17.725773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:17.725904 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:17.726188 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:17.726189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:17.726189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:17.726499 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:17.726649 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:17.727141 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 08:14:17.727275 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260620 08:14:17.727559 28062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907cf1e7747f4ffe9d6d17b2f7da3e45" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37199 } } }
I20260620 08:14:17.727628 28062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:17.728622 28069 catalog_manager.cc:1347] Generated new cluster ID: b34f6de402eb4cf0b6af304a232abd19
I20260620 08:14:17.728677 28069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:17.754254 28069 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:17.755028 28069 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:17.758855 28069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907cf1e7747f4ffe9d6d17b2f7da3e45: Generated new TSK 0
I20260620 08:14:17.759047 28069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:17.797119 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37199
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:17.881726 28081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:17.882025 28081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:17.882053 28081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:17.883272 28081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:17.883313 28081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:17.883340 28081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:17.884799 28081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:17.885052 28081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:17.885296 28081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:17.890362 28089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:17.890378 28087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:17.890362 28086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:17.890462 28081 server_base.cc:1034] running on GCE node
I20260620 08:14:17.890798 28081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:17.891052 28081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:17.892194 28081 hybrid_clock.cc:648] HybridClock initialized: now 1781943257892183 us; error 28 us; skew 500 ppm
Jun 20 08:14:17 dist-test-slave-65w8 krb5kdc[27974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943257, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:17.894910 28081 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:17.895345 28081 webserver.cc:460] Webserver started at http://127.24.249.129:37317/ using document root <none> and password file <none>
I20260620 08:14:17.895570 28081 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:17.895638 28081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:17.895788 28081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:17.896855 28081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "0b48be47840b4500a745ed1b2f5ca5c7"
format_stamp: "Formatted at 2026-06-20 08:14:17 on dist-test-slave-65w8"
server_key: "31daad804240ce61cd2af00b625b7171"
server_key_iv: "f9f10439dd061673669c62a3bd6f0bdc"
server_key_version: "encryptionkey@0"
I20260620 08:14:17.897215 28081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "0b48be47840b4500a745ed1b2f5ca5c7"
format_stamp: "Formatted at 2026-06-20 08:14:17 on dist-test-slave-65w8"
server_key: "31daad804240ce61cd2af00b625b7171"
server_key_iv: "f9f10439dd061673669c62a3bd6f0bdc"
server_key_version: "encryptionkey@0"
I20260620 08:14:17.898653 28081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:17.899456 28096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:17.899693 28081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:17.899801 28081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "0b48be47840b4500a745ed1b2f5ca5c7"
format_stamp: "Formatted at 2026-06-20 08:14:17 on dist-test-slave-65w8"
server_key: "31daad804240ce61cd2af00b625b7171"
server_key_iv: "f9f10439dd061673669c62a3bd6f0bdc"
server_key_version: "encryptionkey@0"
I20260620 08:14:17.899874 28081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:17.912966 28081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:17.915167 28081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:17.915344 28081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:17.915593 28081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:17.915921 28081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:17.915977 28081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:17.916013 28081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:17.916040 28081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:17.922623 28081 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:32995
I20260620 08:14:17.922674 28209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:32995 every 8 connection(s)
I20260620 08:14:17.923038 28081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:14:17 dist-test-slave-65w8 krb5kdc[27974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943257, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:17.932716 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28081
I20260620 08:14:17.932848 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:17.933017 25574 external_mini_cluster.cc:1428] Setting key 1bf087aa686ae44be700da2148715b5b
I20260620 08:14:17.933604 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:17.923822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:40209 (local address 127.24.249.190:37199)
0620 08:14:17.923893 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:17.923895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:17.924197 (+ 302us) server_negotiation.cc:407] Connection header received
0620 08:14:17.924871 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:17.924873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:17.924883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:17.924943 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:17.926077 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:17.926630 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:17.927663 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:17.927875 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:17.927940 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:17.930393 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:17.930395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:17.930395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:17.930405 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:17.931991 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:17.932476 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:17.932477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:17.932477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:17.932538 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:17.932811 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:17.932812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:17.932813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:17.932980 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:17.933055 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:17.933412 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 08:14:17.933515 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:14:17.933768 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:17.923814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:37199 (local address 127.24.249.129:40209)
0620 08:14:17.924039 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:17.924079 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:17.924740 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:17.925053 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:17.925057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:17.925697 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:17.925971 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:17.925977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:17.926730 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:17.926731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:17.927557 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:17.927559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:17.927987 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:17.927988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:17.928103 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:17.928700 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:17.928710 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:17.930273 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:17.932117 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:17.932127 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:17.932130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:17.932381 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:17.932609 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:17.932610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:17.932610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:17.932738 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:17.933059 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:17.933067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:17.933236 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 08:14:17.933364 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 08:14:17.933528 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":114,"threads_started":1}
I20260620 08:14:17.934365 28210 heartbeater.cc:347] Connected to a master server at 127.24.249.190:37199
I20260620 08:14:17.934478 28210 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:17.934690 28210 heartbeater.cc:511] Master 127.24.249.190:37199 requested a full tablet report, sending...
I20260620 08:14:17.935287 28019 ts_manager.cc:194] Registered new tserver with Master: 0b48be47840b4500a745ed1b2f5ca5c7 (127.24.249.129:32995)
I20260620 08:14:17.936257 28019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:40209
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:17.960820 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37199
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:18.044894 28215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:18.045135 28215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:18.045161 28215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:18.046449 28215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:18.046499 28215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:18.046527 28215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:18.047948 28215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:18.048159 28215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:18.048470 28215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:18.053894 28220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:18.054041 28221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:18.054114 28215 server_base.cc:1034] running on GCE node
W20260620 08:14:18.054119 28223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:18.054631 28215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:18.054900 28215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:18.056061 28215 hybrid_clock.cc:648] HybridClock initialized: now 1781943258056040 us; error 35 us; skew 500 ppm
Jun 20 08:14:18 dist-test-slave-65w8 krb5kdc[27974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943258, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:18.058755 28215 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:18.059176 28215 webserver.cc:460] Webserver started at http://127.24.249.130:42187/ using document root <none> and password file <none>
I20260620 08:14:18.059376 28215 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:18.059415 28215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:18.059504 28215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:18.060587 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "16b712102fd74d938d0e2bae2e98b14e"
format_stamp: "Formatted at 2026-06-20 08:14:18 on dist-test-slave-65w8"
server_key: "3e2db0b8cbff8ecd67edcd36ef8e83fb"
server_key_iv: "5a57ff5426fe8e6d506b0168a97cf76a"
server_key_version: "encryptionkey@0"
I20260620 08:14:18.060897 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "16b712102fd74d938d0e2bae2e98b14e"
format_stamp: "Formatted at 2026-06-20 08:14:18 on dist-test-slave-65w8"
server_key: "3e2db0b8cbff8ecd67edcd36ef8e83fb"
server_key_iv: "5a57ff5426fe8e6d506b0168a97cf76a"
server_key_version: "encryptionkey@0"
I20260620 08:14:18.062275 28215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:18.063093 28230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:18.063371 28215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:18.063462 28215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "16b712102fd74d938d0e2bae2e98b14e"
format_stamp: "Formatted at 2026-06-20 08:14:18 on dist-test-slave-65w8"
server_key: "3e2db0b8cbff8ecd67edcd36ef8e83fb"
server_key_iv: "5a57ff5426fe8e6d506b0168a97cf76a"
server_key_version: "encryptionkey@0"
I20260620 08:14:18.063555 28215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:18.083165 28215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:18.085675 28215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:18.085835 28215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:18.086148 28215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:18.086484 28215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:18.086539 28215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:18.086575 28215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:18.086601 28215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:18.092804 28215 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:41553
I20260620 08:14:18.092875 28343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:41553 every 8 connection(s)
I20260620 08:14:18.093261 28215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:18.096403 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28215
I20260620 08:14:18.096489 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:18.096714 25574 external_mini_cluster.cc:1428] Setting key 14079a92e1d5a4e74dc7e71cc5a4a9d1
Jun 20 08:14:18 dist-test-slave-65w8 krb5kdc[27974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943258, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:18.104096 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:18.094095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:54051 (local address 127.24.249.190:37199)
0620 08:14:18.094154 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:18.094156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:18.094407 (+ 251us) server_negotiation.cc:407] Connection header received
0620 08:14:18.094913 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:18.094914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:18.094927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:18.094972 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:18.096000 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.096545 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.097890 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.098103 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.098189 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:18.100623 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:18.100625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:18.100625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:18.100633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:18.102385 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:18.102910 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:18.102912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:18.102913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:18.102988 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:18.103300 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:18.103302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:18.103302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:18.103462 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:18.103550 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:18.103877 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 08:14:18.103975 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:14:18.104211 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:18.094088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:37199 (local address 127.24.249.130:54051)
0620 08:14:18.094260 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:18.094302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:18.094809 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:18.095042 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:18.095051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:18.095690 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:18.095895 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:18.095900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.096622 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.096624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:18.097795 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:18.097797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.098200 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.098201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:18.098338 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:18.098940 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:18.098951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:18.100514 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:18.102491 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:18.102498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:18.102500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:18.102792 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:18.103085 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:18.103086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:18.103086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:18.103220 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:18.103539 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:18.103544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:18.103729 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 08:14:18.103822 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:14:18.104009 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260620 08:14:18.104828 28344 heartbeater.cc:347] Connected to a master server at 127.24.249.190:37199
I20260620 08:14:18.104959 28344 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:18.105158 28344 heartbeater.cc:511] Master 127.24.249.190:37199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:14:18.105558 28019 ts_manager.cc:194] Registered new tserver with Master: 16b712102fd74d938d0e2bae2e98b14e (127.24.249.130:41553)
I20260620 08:14:18.106055 28019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:54051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:18.126973 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37199
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:18.212754 28349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:18.212967 28349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:18.212990 28349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:18.214241 28349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:18.214284 28349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:18.214310 28349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:18.215720 28349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:18.215968 28349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:18.216198 28349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:18.221201 28354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:18.221297 28357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:18.221203 28355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:18.221446 28349 server_base.cc:1034] running on GCE node
I20260620 08:14:18.221642 28349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:18.221933 28349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:18.223104 28349 hybrid_clock.cc:648] HybridClock initialized: now 1781943258223078 us; error 42 us; skew 500 ppm
Jun 20 08:14:18 dist-test-slave-65w8 krb5kdc[27974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943258, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:18.226017 28349 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:18.226464 28349 webserver.cc:460] Webserver started at http://127.24.249.131:35689/ using document root <none> and password file <none>
I20260620 08:14:18.226686 28349 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:18.226732 28349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:18.226876 28349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:18.227994 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "99db09def66240feb6d243e094a2ad14"
format_stamp: "Formatted at 2026-06-20 08:14:18 on dist-test-slave-65w8"
server_key: "73985d1498e73e5cce84728cfb4266b8"
server_key_iv: "1f20ed60b9b99d7f40f90ce7a69f93a7"
server_key_version: "encryptionkey@0"
I20260620 08:14:18.228363 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "99db09def66240feb6d243e094a2ad14"
format_stamp: "Formatted at 2026-06-20 08:14:18 on dist-test-slave-65w8"
server_key: "73985d1498e73e5cce84728cfb4266b8"
server_key_iv: "1f20ed60b9b99d7f40f90ce7a69f93a7"
server_key_version: "encryptionkey@0"
I20260620 08:14:18.229725 28349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:14:18.230557 28364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:18.230747 28349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:18.230846 28349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "99db09def66240feb6d243e094a2ad14"
format_stamp: "Formatted at 2026-06-20 08:14:18 on dist-test-slave-65w8"
server_key: "73985d1498e73e5cce84728cfb4266b8"
server_key_iv: "1f20ed60b9b99d7f40f90ce7a69f93a7"
server_key_version: "encryptionkey@0"
I20260620 08:14:18.230898 28349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:18.260825 28349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:18.263330 28349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:18.263447 28349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:18.263669 28349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:18.264014 28349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:18.264051 28349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:18.264075 28349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:18.264088 28349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:18.270229 28349 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:45409
I20260620 08:14:18.270295 28477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:45409 every 8 connection(s)
I20260620 08:14:18.270617 28349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:18.272653 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28349
I20260620 08:14:18.272758 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:18.272931 25574 external_mini_cluster.cc:1428] Setting key 59b2773eb2cd1476e4ae58a6d1684c92
Jun 20 08:14:18 dist-test-slave-65w8 krb5kdc[27974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943258, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:18.280054 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:18.271376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:48319 (local address 127.24.249.190:37199)
0620 08:14:18.271449 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:18.271451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:18.271906 (+ 455us) server_negotiation.cc:407] Connection header received
0620 08:14:18.272218 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:18.272220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:18.272231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:18.272292 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:18.273525 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.274259 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.275257 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.275439 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.275494 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:18.277617 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:18.277618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:18.277618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:18.277624 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:18.278905 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:18.279283 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:18.279283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:18.279284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:18.279330 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:18.279537 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:18.279538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:18.279538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:18.279634 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:18.279714 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:18.279929 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 08:14:18.279989 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:14:18.280254 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:18.271347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:37199 (local address 127.24.249.131:48319)
0620 08:14:18.271560 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:18.271607 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:18.272104 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:18.272371 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:18.272375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:18.273115 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:18.273420 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:18.273427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.274336 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.274337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:18.275143 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:18.275144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:18.275511 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:18.275512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:18.275610 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:18.276090 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:18.276098 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:18.277511 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:18.278986 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:18.278991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:18.278993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:18.279207 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:18.279385 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:18.279386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:18.279386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:18.279482 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:18.279705 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:18.279709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:18.279819 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 08:14:18.279902 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:14:18.280016 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":110,"threads_started":1}
I20260620 08:14:18.280714 28478 heartbeater.cc:347] Connected to a master server at 127.24.249.190:37199
I20260620 08:14:18.280825 28478 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:18.281001 28478 heartbeater.cc:511] Master 127.24.249.190:37199 requested a full tablet report, sending...
I20260620 08:14:18.281371 28019 ts_manager.cc:194] Registered new tserver with Master: 99db09def66240feb6d243e094a2ad14 (127.24.249.131:45409)
I20260620 08:14:18.281755 28019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:48319
I20260620 08:14:18.284868 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:14:18.296641 28058 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:18.295613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.24.249.190:37199)
0620 08:14:18.295684 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:18.295686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:18.295850 (+ 164us) server_negotiation.cc:407] Connection header received
0620 08:14:18.295954 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:18.295955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:18.295972 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:18.296033 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:18.296554 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36418: BlockingRecv error: recv got EOF from 127.0.0.1:36418 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:14:18.297502 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28081
I20260620 08:14:18.301723 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28215
I20260620 08:14:18.306291 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28349
I20260620 08:14:18.310801 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 27987
2026-06-20T08:14:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1955 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:18 dist-test-slave-65w8 krb5kdc[28490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:18 dist-test-slave-65w8 krb5kdc[28490](info): set up 2 sockets
Jun 20 08:14:18 dist-test-slave-65w8 krb5kdc[28490](info): commencing operation
krb5kdc: starting...
W20260620 08:14:19.362054 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:19 dist-test-slave-65w8 krb5kdc[28490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:19.482316 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:35195
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:35195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:19.565227 28503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:19.565447 28503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:19.565470 28503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:19.566613 28503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:19.566653 28503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:19.566668 28503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:19.566680 28503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:19.566692 28503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:19.568241 28503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:35195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:19.568444 28503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:19.568612 28503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:19.573726 28508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:19.573717 28509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:19.573839 28503 server_base.cc:1034] running on GCE node
W20260620 08:14:19.573721 28511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:19.574323 28503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:19.574616 28503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:19.575783 28503 hybrid_clock.cc:648] HybridClock initialized: now 1781943259575762 us; error 32 us; skew 500 ppm
Jun 20 08:14:19 dist-test-slave-65w8 krb5kdc[28490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943259, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:19.583700 28503 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:19.584192 28503 webserver.cc:460] Webserver started at http://127.24.249.190:43983/ using document root <none> and password file <none>
I20260620 08:14:19.584440 28503 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:19.584510 28503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:19.584657 28503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:19.585959 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2"
format_stamp: "Formatted at 2026-06-20 08:14:19 on dist-test-slave-65w8"
server_key: "b727863da20a0ebdb2d3acda292a532f"
server_key_iv: "42947f5e627d56e6c8a84f790b67a81a"
server_key_version: "encryptionkey@0"
I20260620 08:14:19.586407 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2"
format_stamp: "Formatted at 2026-06-20 08:14:19 on dist-test-slave-65w8"
server_key: "b727863da20a0ebdb2d3acda292a532f"
server_key_iv: "42947f5e627d56e6c8a84f790b67a81a"
server_key_version: "encryptionkey@0"
I20260620 08:14:19.587952 28503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:14:19.588879 28518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:19.589172 28503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:19.589427 28503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2"
format_stamp: "Formatted at 2026-06-20 08:14:19 on dist-test-slave-65w8"
server_key: "b727863da20a0ebdb2d3acda292a532f"
server_key_iv: "42947f5e627d56e6c8a84f790b67a81a"
server_key_version: "encryptionkey@0"
I20260620 08:14:19.589525 28503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:19.598933 28503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:19.601605 28503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:19.601778 28503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:19.606033 28503 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:35195
I20260620 08:14:19.606067 28570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:35195 every 8 connection(s)
I20260620 08:14:19.606570 28503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:19.607300 28571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:19.607671 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28503
I20260620 08:14:19.607769 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:19.607955 25574 external_mini_cluster.cc:1428] Setting key 9d0dac178820249798f986f003007905
I20260620 08:14:19.610260 28571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2: Bootstrap starting.
I20260620 08:14:19.610883 28571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:19.611166 28571 log.cc:824] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:19.611837 28571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2: No bootstrap required, opened a new log
I20260620 08:14:19.613633 28571 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35195 } }
I20260620 08:14:19.613795 28571 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:19.613845 28571 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8dc40b9f9b74eb6aee100ab0eb505a2, State: Initialized, Role: FOLLOWER
I20260620 08:14:19.614008 28571 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35195 } }
I20260620 08:14:19.614090 28571 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:19.614114 28571 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:19.614176 28571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:14:19 dist-test-slave-65w8 krb5kdc[28490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:19.614756 28571 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35195 } }
I20260620 08:14:19.614876 28571 leader_election.cc:302] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8dc40b9f9b74eb6aee100ab0eb505a2; no voters:
I20260620 08:14:19.615069 28571 leader_election.cc:288] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:19.615098 28576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:19.615248 28576 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [term 1 LEADER]: Becoming Leader. State: Replica: b8dc40b9f9b74eb6aee100ab0eb505a2, State: Running, Role: LEADER
I20260620 08:14:19.615330 28576 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35195 } }
I20260620 08:14:19.615361 28571 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:19.615667 28576 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8dc40b9f9b74eb6aee100ab0eb505a2. Latest consensus state: current_term: 1 leader_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35195 } } }
I20260620 08:14:19.615689 28578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8dc40b9f9b74eb6aee100ab0eb505a2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35195 } } }
I20260620 08:14:19.615742 28576 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:19.615767 28578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2 [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:19.617326 28591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:19.617383 28591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:19.617431 28592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:19.617580 28592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:19.618876 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:19.608488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.24.249.190:35195)
0620 08:14:19.608703 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:19.608710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:19.608735 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:14:19.609193 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:19.609200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:19.609266 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:19.609424 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:19.610133 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:19.611148 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:19.611913 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:19.612339 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:19.612461 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:19.614900 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:19.614904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:19.614904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:19.614916 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:19.617114 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:19.617522 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:19.617524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:19.617525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:19.617645 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:19.617939 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:19.617940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:19.617941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:19.618214 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:19.618375 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:19.618561 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 08:14:19.618713 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260620 08:14:19.619391 28592 catalog_manager.cc:1347] Generated new cluster ID: f553b583384b4835a1731cedcf41dc01
I20260620 08:14:19.619439 28592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:19.619817 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:14:19.626797 28592 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:19.627537 28592 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:19.636924 28592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8dc40b9f9b74eb6aee100ab0eb505a2: Generated new TSK 0
I20260620 08:14:19.637144 28592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:14:19.641575 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:19.637037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.24.249.190:35195)
0620 08:14:19.637083 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:19.637085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:19.637458 (+ 373us) server_negotiation.cc:407] Connection header received
0620 08:14:19.637538 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:19.637538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:19.637549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:19.637602 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:19.638120 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:19.638610 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:19.640072 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:19.640291 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:19.640347 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:19.641076 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:19.641077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:19.641077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:19.641262 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:19.641283 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:19.641289 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:19.641332 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:19.641441 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 08:14:19.641467 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:14:19.643954 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28503
2026-06-20T08:14:19Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1330 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:19 dist-test-slave-65w8 krb5kdc[28604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:19 dist-test-slave-65w8 krb5kdc[28604](info): set up 2 sockets
Jun 20 08:14:19 dist-test-slave-65w8 krb5kdc[28604](info): commencing operation
krb5kdc: starting...
W20260620 08:14:20.706157 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:20 dist-test-slave-65w8 krb5kdc[28604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:20.833915 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42001
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:42001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:20.918089 28617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:20.918304 28617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:20.918326 28617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:20.919545 28617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:20.919591 28617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:20.919606 28617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:20.919620 28617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:20.919631 28617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:20.921056 28617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:42001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:20.921264 28617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:20.921439 28617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:20.926398 28622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:20.926426 28623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:20.926443 28617 server_base.cc:1034] running on GCE node
W20260620 08:14:20.926424 28625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:20.927116 28617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:20.927439 28617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:20.928606 28617 hybrid_clock.cc:648] HybridClock initialized: now 1781943260928561 us; error 57 us; skew 500 ppm
Jun 20 08:14:20 dist-test-slave-65w8 krb5kdc[28604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943260, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:20.931205 28617 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:20.931660 28617 webserver.cc:460] Webserver started at http://127.24.249.190:42025/ using document root <none> and password file <none>
I20260620 08:14:20.931921 28617 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:20.931991 28617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:20.932150 28617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:20.933274 28617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "e3d1a39313044c3c9ddde1a5fa65271e"
format_stamp: "Formatted at 2026-06-20 08:14:20 on dist-test-slave-65w8"
server_key: "5c8391423a8135f656aab0d672c2b427"
server_key_iv: "b8009a8c1fc9d72e5ebaab78ef278cd8"
server_key_version: "encryptionkey@0"
I20260620 08:14:20.933634 28617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "e3d1a39313044c3c9ddde1a5fa65271e"
format_stamp: "Formatted at 2026-06-20 08:14:20 on dist-test-slave-65w8"
server_key: "5c8391423a8135f656aab0d672c2b427"
server_key_iv: "b8009a8c1fc9d72e5ebaab78ef278cd8"
server_key_version: "encryptionkey@0"
I20260620 08:14:20.935076 28617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:20.935943 28632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:20.936149 28617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:20.936242 28617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "e3d1a39313044c3c9ddde1a5fa65271e"
format_stamp: "Formatted at 2026-06-20 08:14:20 on dist-test-slave-65w8"
server_key: "5c8391423a8135f656aab0d672c2b427"
server_key_iv: "b8009a8c1fc9d72e5ebaab78ef278cd8"
server_key_version: "encryptionkey@0"
I20260620 08:14:20.936297 28617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:20.960639 28617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:20.963030 28617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:20.963147 28617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:20.967029 28617 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:42001
I20260620 08:14:20.967103 28684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:42001 every 8 connection(s)
I20260620 08:14:20.967448 28617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:20.968636 28685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:20.969498 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28617
I20260620 08:14:20.969655 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:20.969851 25574 external_mini_cluster.cc:1428] Setting key 76a9bb6810ab1fdc7c809afc58e89e0d
I20260620 08:14:20.971971 28685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e: Bootstrap starting.
I20260620 08:14:20.972678 28685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:20.972996 28685 log.cc:824] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:20.973731 28685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e: No bootstrap required, opened a new log
I20260620 08:14:20.975901 28685 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42001 } }
I20260620 08:14:20.976042 28685 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:20.976064 28685 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d1a39313044c3c9ddde1a5fa65271e, State: Initialized, Role: FOLLOWER
I20260620 08:14:20.976222 28685 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42001 } }
I20260620 08:14:20.976293 28685 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:20.976315 28685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:20.976384 28685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:20.976955 28685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42001 } }
Jun 20 08:14:20 dist-test-slave-65w8 krb5kdc[28604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:20.977078 28685 leader_election.cc:302] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3d1a39313044c3c9ddde1a5fa65271e; no voters:
I20260620 08:14:20.977274 28685 leader_election.cc:288] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:20.977301 28690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:20.977420 28690 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [term 1 LEADER]: Becoming Leader. State: Replica: e3d1a39313044c3c9ddde1a5fa65271e, State: Running, Role: LEADER
I20260620 08:14:20.977550 28685 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:20.977557 28690 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42001 } }
I20260620 08:14:20.978030 28691 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42001 } } }
I20260620 08:14:20.978047 28692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d1a39313044c3c9ddde1a5fa65271e. Latest consensus state: current_term: 1 leader_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d1a39313044c3c9ddde1a5fa65271e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42001 } } }
I20260620 08:14:20.978197 28691 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:20.978287 28692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:20.979458 28706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:20.979512 28706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:20.979569 28699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:20.979760 28699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:20.981523 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:20.970447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.24.249.190:42001)
0620 08:14:20.970660 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:20.970670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:20.970714 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:14:20.971222 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:20.971231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:20.971298 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:20.971451 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:20.972091 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:20.973427 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:20.974315 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:20.974960 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:20.975085 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:20.977620 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:20.977624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:20.977624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:20.977637 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:20.979827 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:20.980251 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:20.980255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:20.980255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:20.980369 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:20.980611 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:20.980612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:20.980613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:20.980896 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:20.981054 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:20.981181 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 08:14:20.981322 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260620 08:14:20.981983 28699 catalog_manager.cc:1347] Generated new cluster ID: c2da1b2fd15b464998740c116f2b293d
I20260620 08:14:20.982031 28699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:21.007628 28699 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:21.008208 28699 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:21.012807 28699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3d1a39313044c3c9ddde1a5fa65271e: Generated new TSK 0
I20260620 08:14:21.013049 28699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:21.046988 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42001
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:21.139245 28711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:21.139457 28711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:21.139483 28711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:21.140764 28711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:21.140811 28711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:21.140838 28711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:21.142426 28711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:21.142633 28711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:21.142818 28711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:21.148042 28717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:21.148022 28716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:21.148141 28711 server_base.cc:1034] running on GCE node
W20260620 08:14:21.148022 28719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:21.148483 28711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:21.148773 28711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:21.149942 28711 hybrid_clock.cc:648] HybridClock initialized: now 1781943261149916 us; error 38 us; skew 500 ppm
Jun 20 08:14:21 dist-test-slave-65w8 krb5kdc[28604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943261, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:21.152513 28711 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:21.152997 28711 webserver.cc:460] Webserver started at http://127.24.249.129:43473/ using document root <none> and password file <none>
I20260620 08:14:21.153251 28711 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:21.153324 28711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:21.153468 28711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:21.154599 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "479857c13642416f9dcd2f0ab8c50819"
format_stamp: "Formatted at 2026-06-20 08:14:21 on dist-test-slave-65w8"
server_key: "d5c3eb196012d822fd99f17e86b4a750"
server_key_iv: "d8658f3cadd87ec9506a139e023be994"
server_key_version: "encryptionkey@0"
I20260620 08:14:21.154963 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "479857c13642416f9dcd2f0ab8c50819"
format_stamp: "Formatted at 2026-06-20 08:14:21 on dist-test-slave-65w8"
server_key: "d5c3eb196012d822fd99f17e86b4a750"
server_key_iv: "d8658f3cadd87ec9506a139e023be994"
server_key_version: "encryptionkey@0"
I20260620 08:14:21.156363 28711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:21.157410 28726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:21.157644 28711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:21.157763 28711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "479857c13642416f9dcd2f0ab8c50819"
format_stamp: "Formatted at 2026-06-20 08:14:21 on dist-test-slave-65w8"
server_key: "d5c3eb196012d822fd99f17e86b4a750"
server_key_iv: "d8658f3cadd87ec9506a139e023be994"
server_key_version: "encryptionkey@0"
I20260620 08:14:21.157841 28711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:21.166810 28711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:21.169166 28711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:21.169327 28711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:21.169555 28711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:21.169930 28711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:21.169996 28711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:21.170044 28711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:21.170071 28711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:21.176102 28711 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:36787
I20260620 08:14:21.176152 28839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:36787 every 8 connection(s)
I20260620 08:14:21.176496 28711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:21.182917 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28711
I20260620 08:14:21.183024 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:21.183231 25574 external_mini_cluster.cc:1428] Setting key ffe9c1334a38f208d7b3db54ac9e8d7a
Jun 20 08:14:21 dist-test-slave-65w8 krb5kdc[28604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943261, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:21.187402 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:21.177227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42001 (local address 127.24.249.129:59445)
0620 08:14:21.177420 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:21.177463 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:21.178085 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:21.178404 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:21.178408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:21.179101 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:21.179365 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:21.179371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.180062 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.180063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:21.180897 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:21.180899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.181351 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.181352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:21.181471 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:21.182044 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:21.182053 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:21.183916 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:21.185846 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:21.185855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:21.185858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:21.186129 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:21.186385 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:21.186386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:21.186386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:21.186516 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:21.186855 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:21.186861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:21.186988 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 08:14:21.187079 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 08:14:21.187234 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260620 08:14:21.187407 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:21.177325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:59445 (local address 127.24.249.190:42001)
0620 08:14:21.177373 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:21.177374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:21.177586 (+ 212us) server_negotiation.cc:407] Connection header received
0620 08:14:21.178221 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:21.178222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:21.178236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:21.178333 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:21.179461 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.179960 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.180993 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.181242 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.181310 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:21.184151 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:21.184152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:21.184152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:21.184162 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:21.185736 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:21.186211 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:21.186212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:21.186213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:21.186287 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:21.186600 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:21.186601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:21.186602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:21.186761 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:21.186844 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:21.187152 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 08:14:21.187265 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:14:21.187930 28840 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42001
I20260620 08:14:21.188066 28840 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:21.188362 28840 heartbeater.cc:511] Master 127.24.249.190:42001 requested a full tablet report, sending...
I20260620 08:14:21.188971 28649 ts_manager.cc:194] Registered new tserver with Master: 479857c13642416f9dcd2f0ab8c50819 (127.24.249.129:36787)
I20260620 08:14:21.190073 28649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:59445
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:21.214078 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42001
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:21.299856 28845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:21.300123 28845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:21.300148 28845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:21.301396 28845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:21.301441 28845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:21.301468 28845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:21.303025 28845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:21.303269 28845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:21.303506 28845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:21.309013 28851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:21.309160 28853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:21.309026 28845 server_base.cc:1034] running on GCE node
W20260620 08:14:21.309844 28850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:21.310124 28845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:21.310416 28845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:21.311600 28845 hybrid_clock.cc:648] HybridClock initialized: now 1781943261311602 us; error 49 us; skew 500 ppm
Jun 20 08:14:21 dist-test-slave-65w8 krb5kdc[28604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943261, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:21.314666 28845 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:21.315178 28845 webserver.cc:460] Webserver started at http://127.24.249.130:41685/ using document root <none> and password file <none>
I20260620 08:14:21.315433 28845 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:21.315511 28845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:21.315670 28845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:21.316963 28845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "78dcc50e5fc24ce4b903d09a119a5229"
format_stamp: "Formatted at 2026-06-20 08:14:21 on dist-test-slave-65w8"
server_key: "ed770f3afaac6a79b33b4c1d9823c8ed"
server_key_iv: "552e4358ffd27969c904f867c83f66d0"
server_key_version: "encryptionkey@0"
I20260620 08:14:21.317373 28845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "78dcc50e5fc24ce4b903d09a119a5229"
format_stamp: "Formatted at 2026-06-20 08:14:21 on dist-test-slave-65w8"
server_key: "ed770f3afaac6a79b33b4c1d9823c8ed"
server_key_iv: "552e4358ffd27969c904f867c83f66d0"
server_key_version: "encryptionkey@0"
I20260620 08:14:21.318964 28845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 08:14:21.319788 28860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:21.320026 28845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:21.320128 28845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "78dcc50e5fc24ce4b903d09a119a5229"
format_stamp: "Formatted at 2026-06-20 08:14:21 on dist-test-slave-65w8"
server_key: "ed770f3afaac6a79b33b4c1d9823c8ed"
server_key_iv: "552e4358ffd27969c904f867c83f66d0"
server_key_version: "encryptionkey@0"
I20260620 08:14:21.320206 28845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:21.360926 28845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:21.363909 28845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:21.364069 28845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:21.364342 28845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:21.364749 28845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:21.364794 28845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:21.364817 28845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:21.364830 28845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:21.371114 28845 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:34577
I20260620 08:14:21.371170 28973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:34577 every 8 connection(s)
I20260620 08:14:21.371491 28845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:14:21 dist-test-slave-65w8 krb5kdc[28604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943261, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:21.379998 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28845
I20260620 08:14:21.380120 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:21.380409 25574 external_mini_cluster.cc:1428] Setting key c75d2510d086405399116637b209e2c7
I20260620 08:14:21.382372 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:21.372234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:47195 (local address 127.24.249.190:42001)
0620 08:14:21.372356 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:21.372359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:21.372522 (+ 163us) server_negotiation.cc:407] Connection header received
0620 08:14:21.372996 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:21.372998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:21.373008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:21.373083 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:21.374269 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.374845 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.375943 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.376137 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.376211 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:21.378712 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:21.378714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:21.378715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:21.378724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:21.380480 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:21.381092 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:21.381094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:21.381094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:21.381163 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:21.381577 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:21.381578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:21.381578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:21.381699 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:21.381824 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:21.382205 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 08:14:21.382304 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 08:14:21.382660 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:21.372209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42001 (local address 127.24.249.130:47195)
0620 08:14:21.372364 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:21.372405 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:21.372874 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:21.373200 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:21.373204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:21.373840 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:21.374152 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:21.374158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.374955 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.374956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:21.375838 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:21.375840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:21.376227 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:21.376228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:21.376334 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:21.376843 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:21.376851 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:21.378576 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:21.380599 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:21.380611 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:21.380613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:21.380918 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:21.381338 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:21.381339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:21.381339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:21.381498 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:21.381824 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:21.381830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:21.382042 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 08:14:21.382178 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 08:14:21.382371 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260620 08:14:21.383244 28974 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42001
I20260620 08:14:21.383369 28974 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:21.383584 28974 heartbeater.cc:511] Master 127.24.249.190:42001 requested a full tablet report, sending...
I20260620 08:14:21.384014 28649 ts_manager.cc:194] Registered new tserver with Master: 78dcc50e5fc24ce4b903d09a119a5229 (127.24.249.130:34577)
I20260620 08:14:21.384426 28649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:47195
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
I20260620 08:14:22.191365 28840 heartbeater.cc:503] Master 127.24.249.190:42001 was elected leader, sending a full tablet report...
I20260620 08:14:22.385506 28974 heartbeater.cc:503] Master 127.24.249.190:42001 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
W20260620 08:14:22.676736 25574 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.249.131: real 1.295s user 0.000s sys 0.004s
I20260620 08:14:22.677083 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42001
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:22.760224 28979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:22.760499 28979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:22.760527 28979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:22.761911 28979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:22.761965 28979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:22.761993 28979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:22.763451 28979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:22.763682 28979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:22.763872 28979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:22.769685 28984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:22.769685 28987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:22.769891 28985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:22.769932 28979 server_base.cc:1034] running on GCE node
I20260620 08:14:22.770217 28979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:22.770498 28979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:22.771665 28979 hybrid_clock.cc:648] HybridClock initialized: now 1781943262771609 us; error 74 us; skew 500 ppm
Jun 20 08:14:22 dist-test-slave-65w8 krb5kdc[28604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943262, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:22.774528 28979 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:22.775022 28979 webserver.cc:460] Webserver started at http://127.24.249.131:46063/ using document root <none> and password file <none>
I20260620 08:14:22.775285 28979 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:22.775333 28979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:22.775466 28979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:22.776598 28979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "cce9094c89a1410aa36b6caea4d7d746"
format_stamp: "Formatted at 2026-06-20 08:14:22 on dist-test-slave-65w8"
server_key: "dd2df75f7293aee9a4a36649cad06f65"
server_key_iv: "af308b27da99558467c5f54818a0a5fc"
server_key_version: "encryptionkey@0"
I20260620 08:14:22.776927 28979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "cce9094c89a1410aa36b6caea4d7d746"
format_stamp: "Formatted at 2026-06-20 08:14:22 on dist-test-slave-65w8"
server_key: "dd2df75f7293aee9a4a36649cad06f65"
server_key_iv: "af308b27da99558467c5f54818a0a5fc"
server_key_version: "encryptionkey@0"
I20260620 08:14:22.778532 28979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:22.779407 28994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:22.779623 28979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:22.779735 28979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "cce9094c89a1410aa36b6caea4d7d746"
format_stamp: "Formatted at 2026-06-20 08:14:22 on dist-test-slave-65w8"
server_key: "dd2df75f7293aee9a4a36649cad06f65"
server_key_iv: "af308b27da99558467c5f54818a0a5fc"
server_key_version: "encryptionkey@0"
I20260620 08:14:22.779831 28979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:22.798623 28979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:22.801259 28979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:22.801488 28979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:22.801756 28979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:22.802192 28979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:22.802251 28979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:22.802287 28979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:22.802320 28979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:22.808408 28979 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:45821
I20260620 08:14:22.808454 29107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:45821 every 8 connection(s)
I20260620 08:14:22.808797 28979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:22.812744 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 28979
I20260620 08:14:22.812873 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:22.813078 25574 external_mini_cluster.cc:1428] Setting key f707dd7558b984c38e894c63e0fa454f
Jun 20 08:14:22 dist-test-slave-65w8 krb5kdc[28604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943262, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:22.819062 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:22.809525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:50277 (local address 127.24.249.190:42001)
0620 08:14:22.809694 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:22.809697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:22.809971 (+ 274us) server_negotiation.cc:407] Connection header received
0620 08:14:22.810592 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:22.810593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:22.810609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:22.810714 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:22.811718 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:22.812231 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:22.813580 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:22.813806 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:22.813908 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:22.816029 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:22.816030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:22.816031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:22.816045 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:22.817521 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:22.818055 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:22.818056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:22.818056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:22.818122 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:22.818370 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:22.818371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:22.818371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:22.818495 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:22.818575 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:22.818853 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 08:14:22.818948 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
I20260620 08:14:22.819239 29110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:22.809479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42001 (local address 127.24.249.131:50277)
0620 08:14:22.809704 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:22.809838 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:22.810448 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:22.810814 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:22.810817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:22.811407 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:22.811609 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:22.811614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:22.812330 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:22.812332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:22.813490 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:22.813492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:22.813911 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:22.813912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:22.814050 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:22.814551 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:22.814560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:22.815930 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:22.817668 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:22.817676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:22.817678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:22.817968 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:22.818200 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:22.818200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:22.818201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:22.818305 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:22.818563 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:22.818570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:22.818724 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 08:14:22.818835 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 08:14:22.818994 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
I20260620 08:14:22.819808 29108 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42001
I20260620 08:14:22.819900 29108 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:22.820040 29108 heartbeater.cc:511] Master 127.24.249.190:42001 requested a full tablet report, sending...
I20260620 08:14:22.820436 28649 ts_manager.cc:194] Registered new tserver with Master: cce9094c89a1410aa36b6caea4d7d746 (127.24.249.131:45821)
I20260620 08:14:22.820847 28649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:50277
I20260620 08:14:22.824795 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:14:22 dist-test-slave-65w8 krb5kdc[28604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943262, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 08:14:22 dist-test-slave-65w8 krb5kdc[28604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943262, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:22.848819 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:22.840912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.24.249.190:42001)
0620 08:14:22.841012 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:22.841014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:22.841113 (+ 99us) server_negotiation.cc:407] Connection header received
0620 08:14:22.841242 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:22.841243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:22.841256 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:22.841351 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:22.842017 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:22.842523 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:22.843335 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:22.843550 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:22.843640 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:22.846085 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:22.846087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:22.846087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:22.846102 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:22.847641 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:22.848019 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:22.848020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:22.848021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:22.848072 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:22.848295 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:22.848297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:22.848297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:22.848462 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:22.848555 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:22.848652 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 08:14:22.848724 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 08:14:22.849123 28649 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60780
I20260620 08:14:22.849660 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28711
I20260620 08:14:22.854723 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28845
I20260620 08:14:22.859215 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28979
I20260620 08:14:22.863538 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 28617
2026-06-20T08:14:22Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3222 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:22 dist-test-slave-65w8 krb5kdc[29121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:22 dist-test-slave-65w8 krb5kdc[29121](info): set up 2 sockets
Jun 20 08:14:22 dist-test-slave-65w8 krb5kdc[29121](info): commencing operation
krb5kdc: starting...
W20260620 08:14:23.930063 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:24.057215 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34023
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:24.142527 29134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:24.142795 29134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:24.142839 29134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:24.144091 29134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:24.144163 29134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:24.144187 29134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:24.144209 29134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:24.144238 29134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:24.145828 29134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:24.146143 29134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:24.146377 29134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:24.151255 29139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:24.151360 29140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:24.151425 29134 server_base.cc:1034] running on GCE node
W20260620 08:14:24.151290 29142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:24.151803 29134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:24.152133 29134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:24.153330 29134 hybrid_clock.cc:648] HybridClock initialized: now 1781943264153312 us; error 38 us; skew 500 ppm
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:24.155885 29134 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:24.156322 29134 webserver.cc:460] Webserver started at http://127.24.249.190:37549/ using document root <none> and password file <none>
I20260620 08:14:24.156559 29134 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:24.156605 29134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:24.156740 29134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:24.157801 29134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "8824e55b2fa84a358e54d070f0d0808a"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "02f6fdbd84a2c09bce9bbaed860cc4a6"
server_key_iv: "cf89c89c04a257ad48ecc4a2a11ff586"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.158210 29134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "8824e55b2fa84a358e54d070f0d0808a"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "02f6fdbd84a2c09bce9bbaed860cc4a6"
server_key_iv: "cf89c89c04a257ad48ecc4a2a11ff586"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.159538 29134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:24.160359 29149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.160560 29134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:24.160678 29134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "8824e55b2fa84a358e54d070f0d0808a"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "02f6fdbd84a2c09bce9bbaed860cc4a6"
server_key_iv: "cf89c89c04a257ad48ecc4a2a11ff586"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.160789 29134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:24.170547 29134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:24.172717 29134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:24.172894 29134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:24.176867 29134 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:34023
I20260620 08:14:24.176940 29201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:34023 every 8 connection(s)
I20260620 08:14:24.177461 29134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:24.178316 29202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:24.180703 29202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a: Bootstrap starting.
I20260620 08:14:24.181339 29202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:24.181671 29202 log.cc:824] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:24.182533 29202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a: No bootstrap required, opened a new log
I20260620 08:14:24.182783 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 29134
I20260620 08:14:24.182905 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:24.183100 25574 external_mini_cluster.cc:1428] Setting key 28dcd797ae88eab1e4b190c7ac26ee8c
I20260620 08:14:24.185611 29202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8824e55b2fa84a358e54d070f0d0808a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34023 } }
I20260620 08:14:24.185784 29202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:24.185833 29202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8824e55b2fa84a358e54d070f0d0808a, State: Initialized, Role: FOLLOWER
I20260620 08:14:24.186025 29202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8824e55b2fa84a358e54d070f0d0808a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34023 } }
I20260620 08:14:24.186121 29202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:24.186165 29202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:24.186213 29202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:24.186820 29202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8824e55b2fa84a358e54d070f0d0808a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34023 } }
I20260620 08:14:24.186934 29202 leader_election.cc:302] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8824e55b2fa84a358e54d070f0d0808a; no voters:
I20260620 08:14:24.187100 29202 leader_election.cc:288] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:24.187131 29207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:24.187354 29207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [term 1 LEADER]: Becoming Leader. State: Replica: 8824e55b2fa84a358e54d070f0d0808a, State: Running, Role: LEADER
I20260620 08:14:24.187361 29202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:24.187507 29207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8824e55b2fa84a358e54d070f0d0808a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34023 } }
I20260620 08:14:24.187954 29208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8824e55b2fa84a358e54d070f0d0808a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8824e55b2fa84a358e54d070f0d0808a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34023 } } }
I20260620 08:14:24.188025 29208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:24.188237 29209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8824e55b2fa84a358e54d070f0d0808a. Latest consensus state: current_term: 1 leader_uuid: "8824e55b2fa84a358e54d070f0d0808a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8824e55b2fa84a358e54d070f0d0808a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34023 } } }
I20260620 08:14:24.188339 29209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:24.189375 29223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:24.189435 29223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:24.189481 29213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:24.189626 29213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:24.191074 29213 catalog_manager.cc:1347] Generated new cluster ID: 00659fd004f4444eadd3a9bf51532cf5
I20260620 08:14:24.191125 29213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:24.196310 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.183650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.24.249.190:34023)
0620 08:14:24.183869 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:24.183877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:24.183903 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:14:24.184456 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:24.184463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:24.184511 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:24.184631 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:24.185565 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.186694 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.187788 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.188465 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.188592 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.191550 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:24.191554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:24.191554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:24.191566 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:24.193909 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.194359 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.194362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.194362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.194470 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.194680 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.194681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.194681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.195021 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:24.195184 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:24.195690 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 08:14:24.195833 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260620 08:14:24.210345 29213 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:24.211000 29213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:24.231781 29213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8824e55b2fa84a358e54d070f0d0808a: Generated new TSK 0
I20260620 08:14:24.232015 29213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:24.261500 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34023
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:24.342417 29228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:24.342620 29228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:24.342643 29228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:24.343842 29228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:24.343892 29228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:24.343920 29228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:24.345502 29228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:24.345758 29228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:24.346091 29228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:24.351272 29236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:24.351284 29234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:24.351289 29233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:24.351399 29228 server_base.cc:1034] running on GCE node
I20260620 08:14:24.351689 29228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:24.351889 29228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:24.353072 29228 hybrid_clock.cc:648] HybridClock initialized: now 1781943264353058 us; error 27 us; skew 500 ppm
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:24.355923 29228 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:24.356441 29228 webserver.cc:460] Webserver started at http://127.24.249.129:46131/ using document root <none> and password file <none>
I20260620 08:14:24.356695 29228 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:24.356739 29228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:24.356868 29228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:24.357975 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "873d0f2ee2fc48feb815bb13fb7d02fa"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "d0838610cff150b60cf2e6c073aedd05"
server_key_iv: "d8f2e524f2575ecfd6576cfb95ad02b6"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.358327 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "873d0f2ee2fc48feb815bb13fb7d02fa"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "d0838610cff150b60cf2e6c073aedd05"
server_key_iv: "d8f2e524f2575ecfd6576cfb95ad02b6"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.359685 29228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:24.360469 29243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.360716 29228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:24.360812 29228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "873d0f2ee2fc48feb815bb13fb7d02fa"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "d0838610cff150b60cf2e6c073aedd05"
server_key_iv: "d8f2e524f2575ecfd6576cfb95ad02b6"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.360888 29228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:24.374154 29228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:24.376442 29228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:24.376586 29228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:24.376835 29228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:24.377146 29228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:24.377200 29228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.377236 29228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:24.377262 29228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.383114 29228 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:40581
I20260620 08:14:24.383167 29356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:40581 every 8 connection(s)
I20260620 08:14:24.383466 29228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:24.386755 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 29228
I20260620 08:14:24.386876 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:24.387072 25574 external_mini_cluster.cc:1428] Setting key faa9ac3ae5db7a9c26d8ccea5984f72f
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:24.393927 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.384183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:53511 (local address 127.24.249.190:34023)
0620 08:14:24.384248 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:24.384250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:24.384462 (+ 212us) server_negotiation.cc:407] Connection header received
0620 08:14:24.385376 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:24.385377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:24.385388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:24.385453 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:24.386132 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.386753 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.388092 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.388417 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.388500 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.390960 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:24.390961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:24.390962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:24.390969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:24.392465 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.392902 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.392903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.392904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.392966 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.393229 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.393230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.393230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.393345 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:24.393417 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:24.393699 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 08:14:24.393789 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:14:24.394117 29359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.384152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34023 (local address 127.24.249.129:53511)
0620 08:14:24.384317 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:24.384359 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:24.385244 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:24.385530 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:24.385534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:24.385778 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:24.386032 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:24.386036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.386863 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.386864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:24.387999 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:24.388002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.388514 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.388515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:24.388626 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.389191 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:24.389203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:24.390849 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:24.392555 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:24.392563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:24.392565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:24.392811 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:24.393034 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:24.393035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:24.393035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:24.393152 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:24.393419 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:24.393426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:24.393569 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 08:14:24.393679 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 08:14:24.393835 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260620 08:14:24.394631 29357 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34023
I20260620 08:14:24.394738 29357 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:24.394914 29357 heartbeater.cc:511] Master 127.24.249.190:34023 requested a full tablet report, sending...
I20260620 08:14:24.395551 29166 ts_manager.cc:194] Registered new tserver with Master: 873d0f2ee2fc48feb815bb13fb7d02fa (127.24.249.129:40581)
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:14:24.396397 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:53511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:24.414757 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34023
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:24.496554 29362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:24.496785 29362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:24.496825 29362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:24.498049 29362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:24.498112 29362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:24.498167 29362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:24.499801 29362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:24.500065 29362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:24.500308 29362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:24.506079 29370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:24.506163 29362 server_base.cc:1034] running on GCE node
W20260620 08:14:24.506073 29368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:24.506073 29367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:24.506549 29362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:24.506826 29362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:24.508013 29362 hybrid_clock.cc:648] HybridClock initialized: now 1781943264507991 us; error 44 us; skew 500 ppm
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:24.510797 29362 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:24.511255 29362 webserver.cc:460] Webserver started at http://127.24.249.130:40933/ using document root <none> and password file <none>
I20260620 08:14:24.511502 29362 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:24.511569 29362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:24.511721 29362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:24.512804 29362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "c9389d60f76c44f28cf71ce42e56d2ea"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "790365759b61d07916c1ee1e9260c60b"
server_key_iv: "30e95bae7411899dd4332444ab06196f"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.513163 29362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "c9389d60f76c44f28cf71ce42e56d2ea"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "790365759b61d07916c1ee1e9260c60b"
server_key_iv: "30e95bae7411899dd4332444ab06196f"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.514848 29362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:24.515753 29377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.516023 29362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:24.516139 29362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "c9389d60f76c44f28cf71ce42e56d2ea"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "790365759b61d07916c1ee1e9260c60b"
server_key_iv: "30e95bae7411899dd4332444ab06196f"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.516219 29362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:24.527896 29362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:24.530323 29362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:24.530495 29362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:24.530750 29362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:24.531065 29362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:24.531122 29362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.531158 29362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:24.531205 29362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.537063 29362 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:38945
I20260620 08:14:24.537143 29490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:38945 every 8 connection(s)
I20260620 08:14:24.537475 29362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:24.539994 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 29362
I20260620 08:14:24.540084 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:24.540313 25574 external_mini_cluster.cc:1428] Setting key 53294f5fb14bfa533cebc434b84aec21
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:24.548521 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.538363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:58529 (local address 127.24.249.190:34023)
0620 08:14:24.538443 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:24.538445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:24.538657 (+ 212us) server_negotiation.cc:407] Connection header received
0620 08:14:24.539567 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:24.539568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:24.539579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:24.539644 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:24.540326 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.540884 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.542078 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.542391 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.542474 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.545436 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:24.545438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:24.545439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:24.545447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:24.547080 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.547535 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.547536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.547536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.547597 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.547857 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.547858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.547859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.547979 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:24.548053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:24.548335 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 08:14:24.548426 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:14:24.548734 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.538285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34023 (local address 127.24.249.130:58529)
0620 08:14:24.538463 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:24.538504 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:24.539466 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:24.539698 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:24.539701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:24.539955 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:24.540230 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:24.540235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.541000 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.541001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:24.541998 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:24.542000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.542513 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.542514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:24.542676 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.543351 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:24.543363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:24.545304 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:24.547203 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:24.547211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:24.547213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:24.547454 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:24.547672 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:24.547673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:24.547673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:24.547789 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:24.548044 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:24.548051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:24.548199 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 08:14:24.548311 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 08:14:24.548477 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260620 08:14:24.549196 29491 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34023
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:14:24.549306 29491 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:24.549486 29491 heartbeater.cc:511] Master 127.24.249.190:34023 requested a full tablet report, sending...
I20260620 08:14:24.549906 29166 ts_manager.cc:194] Registered new tserver with Master: c9389d60f76c44f28cf71ce42e56d2ea (127.24.249.130:38945)
I20260620 08:14:24.550462 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:58529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:24.568687 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34023
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:24.648924 29496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:24.649164 29496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:24.649202 29496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:24.650806 29496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:24.650851 29496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:24.650876 29496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:24.652328 29496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:24.652519 29496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:24.652688 29496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:24.657815 29502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:24.657783 29501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:24.657800 29504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:24.658071 29496 server_base.cc:1034] running on GCE node
I20260620 08:14:24.658293 29496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:24.658528 29496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:24.659694 29496 hybrid_clock.cc:648] HybridClock initialized: now 1781943264659669 us; error 46 us; skew 500 ppm
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:24.666265 29496 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:24.666793 29496 webserver.cc:460] Webserver started at http://127.24.249.131:39213/ using document root <none> and password file <none>
I20260620 08:14:24.667049 29496 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:24.667097 29496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:24.667255 29496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:24.668310 29496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "8e6a29fe09bf45ccafb7bdad5b7360ed"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "3aac48c72a41546ae1d89918ea02ee29"
server_key_iv: "3d1fbef2b4c0b5bb4466bff0be16315e"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.668666 29496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "8e6a29fe09bf45ccafb7bdad5b7360ed"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "3aac48c72a41546ae1d89918ea02ee29"
server_key_iv: "3d1fbef2b4c0b5bb4466bff0be16315e"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.670109 29496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:14:24.670907 29511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.671103 29496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:24.671195 29496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "8e6a29fe09bf45ccafb7bdad5b7360ed"
format_stamp: "Formatted at 2026-06-20 08:14:24 on dist-test-slave-65w8"
server_key: "3aac48c72a41546ae1d89918ea02ee29"
server_key_iv: "3d1fbef2b4c0b5bb4466bff0be16315e"
server_key_version: "encryptionkey@0"
I20260620 08:14:24.671293 29496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:24.688215 29496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:24.690574 29496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:24.690692 29496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:24.690953 29496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:24.691282 29496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:24.691315 29496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.691359 29496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:24.691386 29496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:24.697363 29496 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:35947
I20260620 08:14:24.697404 29624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:35947 every 8 connection(s)
I20260620 08:14:24.697741 29496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:24.704069 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 29496
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:24.704182 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:24.704362 25574 external_mini_cluster.cc:1428] Setting key 108662ed006b7e40cbf2b332c028c403
I20260620 08:14:24.707998 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.698539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:33207 (local address 127.24.249.190:34023)
0620 08:14:24.698642 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:24.698645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:24.698810 (+ 165us) server_negotiation.cc:407] Connection header received
0620 08:14:24.699825 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:24.699826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:24.699839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:24.699915 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:24.700607 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.701114 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.702186 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.702384 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.702466 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.704871 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:24.704873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:24.704873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:24.704881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:24.706472 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.707000 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.707002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.707002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.707062 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.707341 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.707342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.707342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.707463 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:24.707541 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:24.707818 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 08:14:24.707900 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 08:14:24.708214 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.698451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34023 (local address 127.24.249.131:33207)
0620 08:14:24.698667 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:24.698721 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:24.699680 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:24.700001 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:24.700005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:24.700304 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:24.700501 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:24.700506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.701206 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.701207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:24.702099 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:24.702101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.702537 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.702538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:24.702650 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.703138 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:24.703146 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:24.704666 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:24.706657 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:24.706671 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:24.706672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:24.706908 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:24.707155 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:24.707156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:24.707156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:24.707269 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:24.707532 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:24.707538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:24.707686 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 08:14:24.707800 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:14:24.707961 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260620 08:14:24.708724 29625 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34023
I20260620 08:14:24.708822 29625 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:24.708966 29625 heartbeater.cc:511] Master 127.24.249.190:34023 requested a full tablet report, sending...
I20260620 08:14:24.709338 29166 ts_manager.cc:194] Registered new tserver with Master: 8e6a29fe09bf45ccafb7bdad5b7360ed (127.24.249.131:35947)
I20260620 08:14:24.709676 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:33207
I20260620 08:14:24.716701 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:24.741530 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.733055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.24.249.129:40581)
0620 08:14:24.733288 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:24.733292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:24.733304 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:24.733331 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:24.733333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:24.733347 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:24.733423 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:24.734246 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.734959 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.735723 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.736057 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.736125 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.738782 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:24.738791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:24.738791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:24.738798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:24.740455 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.740806 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.740808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.740809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.740865 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.741039 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.741040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.741040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.741168 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:24.741276 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:24.741335 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 08:14:24.741396 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:24.749758 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:24.742890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.24.249.190:34023)
0620 08:14:24.742936 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:24.742938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:24.742968 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:14:24.743028 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:24.743028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:24.743037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:24.743084 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:24.743488 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.744017 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.744677 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:24.744879 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:24.744929 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:24.747219 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:24.747222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:24.747222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:24.747229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:24.748640 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.748964 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.748965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.748966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.749015 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:24.749266 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:24.749266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:24.749267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:24.749401 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:24.749475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:24.749599 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 08:14:24.749661 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 08:14:24.750049 29166 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36526
I20260620 08:14:24.750463 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 29228
I20260620 08:14:24.755481 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 29362
I20260620 08:14:24.759802 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 29496
I20260620 08:14:24.764235 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 29134
2026-06-20T08:14:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1900 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29637](info): set up 2 sockets
Jun 20 08:14:24 dist-test-slave-65w8 krb5kdc[29637](info): commencing operation
krb5kdc: starting...
W20260620 08:14:25.822531 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:25 dist-test-slave-65w8 krb5kdc[29637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:25.945330 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:37007
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:37007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:26.031104 29650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:26.031318 29650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:26.031342 29650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:26.032573 29650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:26.032618 29650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:26.032634 29650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:26.032653 29650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:26.032665 29650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:26.034293 29650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:37007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:37007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:26.034548 29650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:26.034787 29650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:26.039932 29658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.039940 29656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.040159 29650 server_base.cc:1034] running on GCE node
W20260620 08:14:26.039932 29655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.040689 29650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:26.041059 29650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:26.042284 29650 hybrid_clock.cc:648] HybridClock initialized: now 1781943266042256 us; error 42 us; skew 500 ppm
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:26.045208 29650 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:26.045583 29650 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:26.045648 29650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:26.045748 29650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:26.046897 29650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "6e1104cb390a43fe9ee8c26c7db4078f"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "5e9754f0b41f4390d68f6e01baf633ab"
server_key_iv: "542e397e46f7dd6423529cbcc115605d"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.047206 29650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "6e1104cb390a43fe9ee8c26c7db4078f"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "5e9754f0b41f4390d68f6e01baf633ab"
server_key_iv: "542e397e46f7dd6423529cbcc115605d"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.048516 29650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:26.049381 29664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.049782 29650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:26.049937 29650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "6e1104cb390a43fe9ee8c26c7db4078f"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "5e9754f0b41f4390d68f6e01baf633ab"
server_key_iv: "542e397e46f7dd6423529cbcc115605d"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.050047 29650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:26.071763 29650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:26.074194 29650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:26.074385 29650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:26.078735 29650 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:37007
I20260620 08:14:26.078761 29716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:37007 every 8 connection(s)
I20260620 08:14:26.079077 29650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:26.080132 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:26.080664 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 29650
I20260620 08:14:26.080780 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:26.080963 25574 external_mini_cluster.cc:1428] Setting key 74bd7eda9e3569bafca5442b90dc1981
I20260620 08:14:26.083024 29717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f: Bootstrap starting.
I20260620 08:14:26.083676 29717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:26.083954 29717 log.cc:824] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:26.084626 29717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f: No bootstrap required, opened a new log
I20260620 08:14:26.086457 29717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37007 } }
I20260620 08:14:26.086639 29717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:26.086711 29717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1104cb390a43fe9ee8c26c7db4078f, State: Initialized, Role: FOLLOWER
I20260620 08:14:26.086864 29717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37007 } }
I20260620 08:14:26.086951 29717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:26.086995 29717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:26.087041 29717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:26.087782 29717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37007 } }
I20260620 08:14:26.087898 29717 leader_election.cc:302] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e1104cb390a43fe9ee8c26c7db4078f; no voters:
I20260620 08:14:26.088058 29717 leader_election.cc:288] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:26.088133 29722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:26.088315 29717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:26.088331 29722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [term 1 LEADER]: Becoming Leader. State: Replica: 6e1104cb390a43fe9ee8c26c7db4078f, State: Running, Role: LEADER
I20260620 08:14:26.089277 29722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37007 } }
I20260620 08:14:26.089987 29724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37007 } } }
I20260620 08:14:26.090018 29723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1104cb390a43fe9ee8c26c7db4078f. Latest consensus state: current_term: 1 leader_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1104cb390a43fe9ee8c26c7db4078f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37007 } } }
I20260620 08:14:26.090250 29724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:26.090263 29723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:26.090749 29736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:26.090804 29736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:26.090885 29739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:26.091100 29739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:26.092759 29739 catalog_manager.cc:1347] Generated new cluster ID: 5e55f68a68774acb8a4309b7b92c2fcb
I20260620 08:14:26.092815 29739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:26.093362 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.081525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.24.249.190:37007)
0620 08:14:26.081745 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.081755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.081790 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:14:26.082297 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.082304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.082353 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.082509 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:26.083160 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.084182 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.085051 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.085500 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.085634 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.088588 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:26.088592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:26.088593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:26.088605 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:26.091248 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.091726 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.091729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.091729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.091831 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.092057 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.092058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.092058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.092360 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:26.092488 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.093011 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.093163 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260620 08:14:26.104707 29739 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:26.105453 29739 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:26.116051 29739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f: Generated new TSK 0
I20260620 08:14:26.116287 29739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:26.146835 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37007
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:26.228892 29743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:26.229148 29743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:26.229187 29743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:26.230386 29743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:26.230458 29743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:26.230504 29743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:26.232054 29743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:26.232340 29743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:26.232583 29743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:26.237958 29751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.237953 29749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.238072 29748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.238204 29743 server_base.cc:1034] running on GCE node
I20260620 08:14:26.238457 29743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:26.238723 29743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:26.239908 29743 hybrid_clock.cc:648] HybridClock initialized: now 1781943266239892 us; error 29 us; skew 500 ppm
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:26.242820 29743 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:26.243201 29743 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:26.243252 29743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:26.243433 29743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:26.244520 29743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "c88a2191db3d4b40a09c0a7f7a5445f2"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "fe4ce72bde70582af517d0511bbb4361"
server_key_iv: "ff8dd9823793982fd932a31f24061a10"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.244882 29743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "c88a2191db3d4b40a09c0a7f7a5445f2"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "fe4ce72bde70582af517d0511bbb4361"
server_key_iv: "ff8dd9823793982fd932a31f24061a10"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.246411 29743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:26.247251 29757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.247493 29743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:26.247584 29743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "c88a2191db3d4b40a09c0a7f7a5445f2"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "fe4ce72bde70582af517d0511bbb4361"
server_key_iv: "ff8dd9823793982fd932a31f24061a10"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.247649 29743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:26.261381 29743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:26.263826 29743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:26.263952 29743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:26.264189 29743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:26.264478 29743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:26.264511 29743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.264536 29743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:26.264550 29743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.270606 29743 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:39979
I20260620 08:14:26.270640 29870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:39979 every 8 connection(s)
I20260620 08:14:26.270983 29743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:26.272459 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 29743
I20260620 08:14:26.272574 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:26.272818 25574 external_mini_cluster.cc:1428] Setting key d466cd01f45a7200df3dfa7b3191694b
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:14:26.281368 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.271691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:35781 (local address 127.24.249.190:37007)
0620 08:14:26.271736 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.271738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.272000 (+ 262us) server_negotiation.cc:407] Connection header received
0620 08:14:26.272596 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.272598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.272613 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.272686 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:26.274010 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.274503 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.275523 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.275748 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.275839 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.278300 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:26.278302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:26.278302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:26.278311 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:26.279804 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.280313 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.280315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.280315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.280381 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.280659 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.280660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.280660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.280786 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:26.280871 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.281179 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.281270 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:26.281571 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.271661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:37007 (local address 127.24.249.129:35781)
0620 08:14:26.271852 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.271895 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.272461 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.272814 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.272818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.273225 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:26.273830 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.273835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.274614 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.274616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.275427 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.275429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.275849 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.275850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.275976 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.276502 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:26.276510 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:26.278184 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:26.279913 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:26.279921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:26.279923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:26.280209 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:26.280454 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:26.280456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:26.280456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:26.280578 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:26.280861 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:26.280868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:26.281018 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.281152 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.281319 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260620 08:14:26.282197 29871 heartbeater.cc:347] Connected to a master server at 127.24.249.190:37007
I20260620 08:14:26.282306 29871 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:26.282471 29871 heartbeater.cc:511] Master 127.24.249.190:37007 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
I20260620 08:14:26.283152 29681 ts_manager.cc:194] Registered new tserver with Master: c88a2191db3d4b40a09c0a7f7a5445f2 (127.24.249.129:39979)
I20260620 08:14:26.283849 29681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:35781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:26.297341 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37007
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:26.377784 29876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:26.378003 29876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:26.378027 29876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:26.379202 29876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:26.379254 29876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:26.379282 29876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:26.380788 29876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:26.381028 29876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:26.381275 29876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:26.386427 29881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.386427 29882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.386484 29884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.386842 29876 server_base.cc:1034] running on GCE node
I20260620 08:14:26.387089 29876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:26.387346 29876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:26.388522 29876 hybrid_clock.cc:648] HybridClock initialized: now 1781943266388508 us; error 29 us; skew 500 ppm
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:26.391391 29876 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:26.391778 29876 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:26.391865 29876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:26.392050 29876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:26.393260 29876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "d08a68f60c5c40a6af978f073c58ba4b"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "714dc9abfc00b64fc02db4ab6bdc201e"
server_key_iv: "2a8a71ea25c8155bb7b1dc957cd80b3f"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.393584 29876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "d08a68f60c5c40a6af978f073c58ba4b"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "714dc9abfc00b64fc02db4ab6bdc201e"
server_key_iv: "2a8a71ea25c8155bb7b1dc957cd80b3f"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.395058 29876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:26.395957 29890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.396241 29876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:26.396392 29876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "d08a68f60c5c40a6af978f073c58ba4b"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "714dc9abfc00b64fc02db4ab6bdc201e"
server_key_iv: "2a8a71ea25c8155bb7b1dc957cd80b3f"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.396490 29876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:26.424417 29876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:26.426707 29876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:26.426868 29876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:26.427137 29876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:26.427479 29876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:26.427534 29876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.427579 29876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:26.427603 29876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.433431 29876 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:46867
I20260620 08:14:26.433470 30003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:46867 every 8 connection(s)
I20260620 08:14:26.433775 29876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:26.442812 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 29876
I20260620 08:14:26.442946 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:26.443174 25574 external_mini_cluster.cc:1428] Setting key 5b67e381d62a9c65ea079e8141f60a34
I20260620 08:14:26.443838 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.434596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:45265 (local address 127.24.249.190:37007)
0620 08:14:26.434675 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.434677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.434886 (+ 209us) server_negotiation.cc:407] Connection header received
0620 08:14:26.435386 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.435387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.435398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.435458 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:26.436455 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.436980 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.438060 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.438266 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.438329 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.440597 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:26.440598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:26.440599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:26.440606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:26.442217 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.442618 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.442619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.442620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.442685 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.443069 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.443070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.443070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.443246 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:26.443315 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.443660 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.443749 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:14:26.444061 30006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.434563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:37007 (local address 127.24.249.130:45265)
0620 08:14:26.434745 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.434787 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.435267 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.435535 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.435539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.435794 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:26.436330 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.436337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.437078 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.437079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.437964 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.437967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.438345 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.438345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.438442 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.438994 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:26.439002 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:26.440486 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:26.442303 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:26.442310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:26.442312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:26.442525 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:26.442828 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:26.442829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:26.442829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:26.442969 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:26.443307 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:26.443313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:26.443485 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.443626 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.443788 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260620 08:14:26.444689 30004 heartbeater.cc:347] Connected to a master server at 127.24.249.190:37007
I20260620 08:14:26.444768 30004 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:26.444932 30004 heartbeater.cc:511] Master 127.24.249.190:37007 requested a full tablet report, sending...
I20260620 08:14:26.445303 29681 ts_manager.cc:194] Registered new tserver with Master: d08a68f60c5c40a6af978f073c58ba4b (127.24.249.130:46867)
I20260620 08:14:26.445842 29681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:45265
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:26.469456 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37007
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:26.552821 30009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:26.553053 30009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:26.553090 30009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:26.554479 30009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:26.554544 30009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:26.554589 30009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:26.556164 30009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:37007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:26.556435 30009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:26.556670 30009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:14:26.561707 30009 server_base.cc:1034] running on GCE node
W20260620 08:14:26.561673 30015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.561688 30014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.561854 30017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.562227 30009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:26.562477 30009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:26.563654 30009 hybrid_clock.cc:648] HybridClock initialized: now 1781943266563632 us; error 33 us; skew 500 ppm
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:26.566273 30009 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:26.566637 30009 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:26.566687 30009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:26.566776 30009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:26.567819 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "a1288e5675e54a698425d45fd68592d7"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "5dfa69fe85bef3ec8ab790d22dde7202"
server_key_iv: "107baadaff201262fe70299b7921dcdd"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.568125 30009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "a1288e5675e54a698425d45fd68592d7"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "5dfa69fe85bef3ec8ab790d22dde7202"
server_key_iv: "107baadaff201262fe70299b7921dcdd"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.569407 30009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:14:26.570272 30023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.570508 30009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:26.570585 30009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "a1288e5675e54a698425d45fd68592d7"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "5dfa69fe85bef3ec8ab790d22dde7202"
server_key_iv: "107baadaff201262fe70299b7921dcdd"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.570716 30009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:26.586421 30009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:26.588722 30009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:26.588881 30009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:26.589151 30009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:26.589525 30009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:26.589587 30009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.589632 30009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:26.589658 30009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.595758 30009 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:32933
I20260620 08:14:26.595800 30136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:32933 every 8 connection(s)
I20260620 08:14:26.596060 30009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:14:26 dist-test-slave-65w8 krb5kdc[29637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943266, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:26.604750 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30009
I20260620 08:14:26.604833 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:26.605024 25574 external_mini_cluster.cc:1428] Setting key 77d043d4af94d9c6a09dbaf807f45828
I20260620 08:14:26.605255 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.596751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:60909 (local address 127.24.249.190:37007)
0620 08:14:26.596829 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.596831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.596995 (+ 164us) server_negotiation.cc:407] Connection header received
0620 08:14:26.597458 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.597459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.597469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.597520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:26.598470 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.598969 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.599973 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.600165 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.600225 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.602478 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:26.602479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:26.602480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:26.602488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:26.603773 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.604222 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.604223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.604224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.604289 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.604536 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.604537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.604537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.604665 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:26.604745 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.605082 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.605162 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 08:14:26.605454 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.596698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:37007 (local address 127.24.249.131:60909)
0620 08:14:26.596881 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.596916 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.597379 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.597570 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.597574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.597822 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:26.598368 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.598374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.599047 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.599047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.599898 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.599900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.600271 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.600273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.600385 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.600977 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:26.600985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:26.602373 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:26.603884 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:26.603891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:26.603893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:26.604141 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:26.604366 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:26.604366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:26.604367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:26.604470 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:26.604735 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:26.604741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:26.604933 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.605051 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.605219 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260620 08:14:26.605898 30137 heartbeater.cc:347] Connected to a master server at 127.24.249.190:37007
I20260620 08:14:26.605998 30137 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:26.606148 30137 heartbeater.cc:511] Master 127.24.249.190:37007 requested a full tablet report, sending...
I20260620 08:14:26.606521 29680 ts_manager.cc:194] Registered new tserver with Master: a1288e5675e54a698425d45fd68592d7 (127.24.249.131:32933)
I20260620 08:14:26.606909 29680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:60909
I20260620 08:14:26.617115 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:14:26.625316 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.618439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.24.249.190:37007)
0620 08:14:26.618532 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.618534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.618648 (+ 114us) server_negotiation.cc:407] Connection header received
0620 08:14:26.618714 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.618716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.618728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.618818 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:26.619471 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.620032 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.620850 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.621064 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.621157 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.622566 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:26.622568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:26.622568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:26.622584 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:26.624051 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.624464 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.624465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.624466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.624527 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:26.624727 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:26.624728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:26.624728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:26.624889 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:26.625017 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.625126 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.625210 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:14:26.627455 29680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:26.629364 29680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:26.636163 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.632005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.24.249.131:32933)
0620 08:14:26.632232 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.632237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.632397 (+ 160us) server_negotiation.cc:407] Connection header received
0620 08:14:26.632430 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.632431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.632444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.632522 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:26.632884 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.633674 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.635002 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.635707 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.635787 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.635842 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.636027 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.636074 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260620 08:14:26.636271 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.631759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39979 (local address 127.0.0.1:59786)
0620 08:14:26.632193 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.632235 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.632324 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.632530 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.632533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.632536 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:26.632617 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.632619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.633969 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.633970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.635129 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.635131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.636003 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.636004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.636066 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.636070 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.636135 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.636195 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":6,"thread_start_us":63,"threads_started":1}
I20260620 08:14:26.636377 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.631770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46867 (local address 127.0.0.1:42208)
0620 08:14:26.632630 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.632641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.632693 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.632979 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.632980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.632981 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:26.633020 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.633021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.634273 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.634274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.635316 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.635318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.636125 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.636126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.636185 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.636190 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.636239 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.636286 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":87,"threads_started":1}
I20260620 08:14:26.636595 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.631970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.24.249.129:39979)
0620 08:14:26.632144 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.632149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.632312 (+ 163us) server_negotiation.cc:407] Connection header received
0620 08:14:26.632402 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.632405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.632420 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.632525 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:26.632693 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.633806 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.635217 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.635918 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.636005 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.636053 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.636149 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.636190 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260620 08:14:26.636668 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.632028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.24.249.130:46867)
0620 08:14:26.632304 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.632308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.632766 (+ 458us) server_negotiation.cc:407] Connection header received
0620 08:14:26.632810 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.632813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.632828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.632896 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:26.633101 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.634173 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.635396 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.635981 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.636043 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.636092 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.636550 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.636589 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
I20260620 08:14:26.636778 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.631759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:32933 (local address 127.0.0.1:59092)
0620 08:14:26.632191 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.632233 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.632312 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.632531 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.632537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.632540 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:26.632792 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.632795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.633772 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.633773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.634919 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.634922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.635798 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.635799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.635887 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.635901 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.636547 (+ 646us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.636587 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":182,"thread_start_us":98,"threads_started":1}
I20260620 08:14:26.637368 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 971160ea1e7a436ca1be494003b2532b (DEFAULT_TABLE table=test-table [id=876947e7e55645cda9b99d2d69114176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:26.637391 30071 tablet_service.cc:1467] Processing CreateTablet for tablet 971160ea1e7a436ca1be494003b2532b (DEFAULT_TABLE table=test-table [id=876947e7e55645cda9b99d2d69114176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:26.637708 30071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971160ea1e7a436ca1be494003b2532b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:26.637712 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971160ea1e7a436ca1be494003b2532b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:26.640352 30154 tablet_bootstrap.cc:492] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Bootstrap starting.
I20260620 08:14:26.641017 30154 tablet_bootstrap.cc:654] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:26.640913 29938 tablet_service.cc:1467] Processing CreateTablet for tablet 971160ea1e7a436ca1be494003b2532b (DEFAULT_TABLE table=test-table [id=876947e7e55645cda9b99d2d69114176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:26.641247 29938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971160ea1e7a436ca1be494003b2532b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:26.641608 30154 log.cc:824] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:26.642439 30154 tablet_bootstrap.cc:492] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: No bootstrap required, opened a new log
I20260620 08:14:26.642515 30154 ts_tablet_manager.cc:1397] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:26.644584 30157 tablet_bootstrap.cc:492] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: Bootstrap starting.
I20260620 08:14:26.644599 30154 raft_consensus.cc:348] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.644783 30154 raft_consensus.cc:374] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:26.644831 30154 raft_consensus.cc:729] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1288e5675e54a698425d45fd68592d7, State: Initialized, Role: FOLLOWER
I20260620 08:14:26.644924 30155 tablet_bootstrap.cc:492] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2: Bootstrap starting.
I20260620 08:14:26.644948 30154 consensus_queue.cc:260] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.645247 30137 heartbeater.cc:503] Master 127.24.249.190:37007 was elected leader, sending a full tablet report...
I20260620 08:14:26.645254 30154 ts_tablet_manager.cc:1428] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:14:26.645545 30155 tablet_bootstrap.cc:654] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:26.645763 30157 tablet_bootstrap.cc:654] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:26.646412 30157 log.cc:824] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:26.646520 30155 log.cc:824] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:26.647181 30157 tablet_bootstrap.cc:492] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: No bootstrap required, opened a new log
I20260620 08:14:26.647264 30157 ts_tablet_manager.cc:1397] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 08:14:26.647300 30155 tablet_bootstrap.cc:492] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2: No bootstrap required, opened a new log
I20260620 08:14:26.647369 30155 ts_tablet_manager.cc:1397] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:26.648600 30157 raft_consensus.cc:348] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.648766 30157 raft_consensus.cc:374] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:26.648821 30157 raft_consensus.cc:729] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08a68f60c5c40a6af978f073c58ba4b, State: Initialized, Role: FOLLOWER
I20260620 08:14:26.648762 30155 raft_consensus.cc:348] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.648941 30157 consensus_queue.cc:260] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.648977 30155 raft_consensus.cc:374] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:26.649020 30155 raft_consensus.cc:729] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88a2191db3d4b40a09c0a7f7a5445f2, State: Initialized, Role: FOLLOWER
I20260620 08:14:26.649142 30155 consensus_queue.cc:260] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.649242 30157 ts_tablet_manager.cc:1428] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:14:26.649358 30004 heartbeater.cc:503] Master 127.24.249.190:37007 was elected leader, sending a full tablet report...
I20260620 08:14:26.649420 29871 heartbeater.cc:503] Master 127.24.249.190:37007 was elected leader, sending a full tablet report...
I20260620 08:14:26.649665 30155 ts_tablet_manager.cc:1428] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:26.674259 30158 raft_consensus.cc:482] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:26.674407 30158 raft_consensus.cc:504] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.674767 30158 leader_election.cc:288] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c88a2191db3d4b40a09c0a7f7a5445f2 (127.24.249.129:39979), d08a68f60c5c40a6af978f073c58ba4b (127.24.249.130:46867)
I20260620 08:14:26.678738 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.674944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46867 (local address 127.24.249.131:37151)
0620 08:14:26.674999 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.675011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.675051 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.675229 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.675229 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.675232 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:26.675351 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.675353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.676320 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.676321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.677578 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.677581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.678415 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.678416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.678517 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.678524 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.678596 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.678647 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":9}
I20260620 08:14:26.678753 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.675021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:37151 (local address 127.24.249.130:46867)
0620 08:14:26.675067 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.675070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.675080 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:26.675117 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.675118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.675128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.675213 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:26.675431 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.676217 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.677692 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.678298 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.678387 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.678394 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.678615 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.678687 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:26.678752 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.674955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39979 (local address 127.24.249.131:42847)
0620 08:14:26.675127 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.675135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:26.675209 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:26.675353 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:26.675353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:26.675355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:26.675418 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.675420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.676320 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.676321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.677578 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:26.677581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.678415 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.678416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:26.678517 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.678524 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 08:14:26.678635 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 08:14:26.678664 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":78,"thread_start_us":113,"threads_started":1}
I20260620 08:14:26.678787 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.675050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:42847 (local address 127.24.249.129:39979)
0620 08:14:26.675094 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.675096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.675208 (+ 112us) server_negotiation.cc:407] Connection header received
0620 08:14:26.675275 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.675280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.675289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.675333 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:26.675487 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.676206 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.677692 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.678317 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.678385 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.678394 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.678683 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.678721 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:26.679123 29958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971160ea1e7a436ca1be494003b2532b" candidate_uuid: "a1288e5675e54a698425d45fd68592d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08a68f60c5c40a6af978f073c58ba4b" is_pre_election: true
I20260620 08:14:26.679117 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971160ea1e7a436ca1be494003b2532b" candidate_uuid: "a1288e5675e54a698425d45fd68592d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" is_pre_election: true
I20260620 08:14:26.679286 29958 raft_consensus.cc:2454] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1288e5675e54a698425d45fd68592d7 in term 0.
I20260620 08:14:26.679360 29825 raft_consensus.cc:2454] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1288e5675e54a698425d45fd68592d7 in term 0.
I20260620 08:14:26.679610 30024 leader_election.cc:302] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1288e5675e54a698425d45fd68592d7, c88a2191db3d4b40a09c0a7f7a5445f2; no voters:
I20260620 08:14:26.679790 30158 raft_consensus.cc:2781] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:26.679859 30158 raft_consensus.cc:482] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:26.679877 30158 raft_consensus.cc:3037] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:26.680639 30158 raft_consensus.cc:504] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.680781 30158 leader_election.cc:288] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [CANDIDATE]: Term 1 election: Requested vote from peers c88a2191db3d4b40a09c0a7f7a5445f2 (127.24.249.129:39979), d08a68f60c5c40a6af978f073c58ba4b (127.24.249.130:46867)
I20260620 08:14:26.680932 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971160ea1e7a436ca1be494003b2532b" candidate_uuid: "a1288e5675e54a698425d45fd68592d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2"
I20260620 08:14:26.680945 29958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971160ea1e7a436ca1be494003b2532b" candidate_uuid: "a1288e5675e54a698425d45fd68592d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08a68f60c5c40a6af978f073c58ba4b"
I20260620 08:14:26.681030 29825 raft_consensus.cc:3037] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:26.681025 29958 raft_consensus.cc:3037] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:26.682067 29958 raft_consensus.cc:2454] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1288e5675e54a698425d45fd68592d7 in term 1.
I20260620 08:14:26.682137 29825 raft_consensus.cc:2454] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1288e5675e54a698425d45fd68592d7 in term 1.
I20260620 08:14:26.682276 30025 leader_election.cc:302] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1288e5675e54a698425d45fd68592d7, d08a68f60c5c40a6af978f073c58ba4b; no voters:
I20260620 08:14:26.682361 30158 raft_consensus.cc:2781] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:26.682516 30158 raft_consensus.cc:686] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 1 LEADER]: Becoming Leader. State: Replica: a1288e5675e54a698425d45fd68592d7, State: Running, Role: LEADER
I20260620 08:14:26.682618 30158 consensus_queue.cc:237] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } }
I20260620 08:14:26.683480 29680 catalog_manager.cc:5495] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1288e5675e54a698425d45fd68592d7 (127.24.249.131). New cstate: current_term: 1 leader_uuid: "a1288e5675e54a698425d45fd68592d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1288e5675e54a698425d45fd68592d7" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 32933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 } health_report { overall_health: UNKNOWN } } }
W20260620 08:14:26.684356 30005 tablet.cc:2369] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:26.696389 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.693931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.24.249.131:32933)
0620 08:14:26.693991 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.693994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.694002 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:26.694051 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.694052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.694068 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.694154 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:26.694306 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.694788 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.695573 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:26.695774 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:26.695850 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:26.696021 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:26.696130 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:26.696228 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.696254 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.696273 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:26.698294 29958 raft_consensus.cc:1261] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 1 FOLLOWER]: Refusing update from remote peer a1288e5675e54a698425d45fd68592d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:26.698307 29825 raft_consensus.cc:1261] T 971160ea1e7a436ca1be494003b2532b P c88a2191db3d4b40a09c0a7f7a5445f2 [term 1 FOLLOWER]: Refusing update from remote peer a1288e5675e54a698425d45fd68592d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:26.698725 30158 consensus_queue.cc:1035] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08a68f60c5c40a6af978f073c58ba4b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:26.698892 30164 consensus_queue.cc:1035] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c88a2191db3d4b40a09c0a7f7a5445f2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:26.701654 30169 mvcc.cc:205] Tried to move back new op lower bound from 7298839620393357312 to 7298839620332699648. Current Snapshot: MvccSnapshot[applied={T|T < 7298839620393357312}]
I20260620 08:14:26.703684 29680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:26.703785 29680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:26.704603 29680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e1104cb390a43fe9ee8c26c7db4078f: Sending DeleteTablet for 3 replicas of tablet 971160ea1e7a436ca1be494003b2532b
I20260620 08:14:26.704928 30071 tablet_service.cc:1514] Processing DeleteTablet for tablet 971160ea1e7a436ca1be494003b2532b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:26 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:59092
I20260620 08:14:26.704967 29938 tablet_service.cc:1514] Processing DeleteTablet for tablet 971160ea1e7a436ca1be494003b2532b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:26 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:42208
I20260620 08:14:26.705240 30176 tablet_replica.cc:330] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: stopping tablet replica
I20260620 08:14:26.705207 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 971160ea1e7a436ca1be494003b2532b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:26 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:59786
I20260620 08:14:26.705338 30176 raft_consensus.cc:2229] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:26.705482 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 29743
I20260620 08:14:26.706789 30176 raft_consensus.cc:2258] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:26.707332 30176 ts_tablet_manager.cc:1905] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:26.709442 30176 ts_tablet_manager.cc:1918] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:26.709523 30176 log.cc:1196] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/971160ea1e7a436ca1be494003b2532b
I20260620 08:14:26.709793 30176 ts_tablet_manager.cc:1939] T 971160ea1e7a436ca1be494003b2532b P a1288e5675e54a698425d45fd68592d7: Deleting consensus metadata
I20260620 08:14:26.710500 29667 catalog_manager.cc:4841] TS a1288e5675e54a698425d45fd68592d7 (127.24.249.131:32933): tablet 971160ea1e7a436ca1be494003b2532b (table test-table [id=876947e7e55645cda9b99d2d69114176]) successfully deleted
W20260620 08:14:26.712025 29665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:39979 (error 108)
I20260620 08:14:26.712090 30177 tablet_replica.cc:330] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: stopping tablet replica
I20260620 08:14:26.712246 30177 raft_consensus.cc:2229] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:26.712366 30177 raft_consensus.cc:2258] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:26.712863 30177 ts_tablet_manager.cc:1905] T 971160ea1e7a436ca1be494003b2532b P d08a68f60c5c40a6af978f073c58ba4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:26.712935 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 29876
I20260620 08:14:26.713065 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.712924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39979 (local address 127.0.0.1:59800)
0620 08:14:26.712977 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.713025 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:39979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 08:14:26.713709 29665 catalog_manager.cc:4584] TS c88a2191db3d4b40a09c0a7f7a5445f2 (127.24.249.129:39979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 971160ea1e7a436ca1be494003b2532b: Network error: Client connection negotiation failed: client connection to 127.24.249.129:39979: connect: Connection refused (error 111)
I20260620 08:14:26.717916 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.717754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46867 (local address 127.0.0.1:42210)
0620 08:14:26.717814 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:26.717849 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.130:46867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:14:26.717903 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30009
I20260620 08:14:26.723290 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 29650
2026-06-20T08:14:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1959 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T08:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:26Z Disabled control of system clock
I20260620 08:14:26.752864 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41397
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:14:26.834167 30183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:26.834375 30183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:26.834398 30183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:26.835582 30183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:26.835624 30183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:26.835636 30183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:26.835649 30183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:26.835661 30183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:26.837270 30183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:41397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:26.837601 30183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:26.837989 30183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:26.843042 30191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.843086 30188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.843037 30189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.843412 30183 server_base.cc:1034] running on GCE node
I20260620 08:14:26.843726 30183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:26.844040 30183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:26.845209 30183 hybrid_clock.cc:648] HybridClock initialized: now 1781943266845189 us; error 64 us; skew 500 ppm
I20260620 08:14:26.846621 30183 webserver.cc:460] Webserver started at http://127.24.249.190:39559/ using document root <none> and password file <none>
I20260620 08:14:26.846951 30183 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:26.847002 30183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:26.847097 30183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:26.848173 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "3f70799840b3490494f7e652fa41e133"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "7c78ae5dbea6589bca04d50a9a8b6301"
server_key_iv: "3ee6d93800f91f1328e2c2675da3f032"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.848472 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "3f70799840b3490494f7e652fa41e133"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "7c78ae5dbea6589bca04d50a9a8b6301"
server_key_iv: "3ee6d93800f91f1328e2c2675da3f032"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.849701 30183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:26.850628 30197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.850878 30183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:26.850970 30183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "3f70799840b3490494f7e652fa41e133"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "7c78ae5dbea6589bca04d50a9a8b6301"
server_key_iv: "3ee6d93800f91f1328e2c2675da3f032"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.851042 30183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:26.854350 30183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:26.854643 30183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:26.854791 30183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:26.858548 30183 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:41397
I20260620 08:14:26.858662 30249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:41397 every 8 connection(s)
I20260620 08:14:26.858954 30183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:26.859666 30250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:26.862514 30250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133: Bootstrap starting.
I20260620 08:14:26.863107 30250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:26.863406 30250 log.cc:824] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:26.864177 30250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133: No bootstrap required, opened a new log
I20260620 08:14:26.865837 30250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70799840b3490494f7e652fa41e133" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41397 } }
I20260620 08:14:26.866109 30250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:26.866159 30250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f70799840b3490494f7e652fa41e133, State: Initialized, Role: FOLLOWER
I20260620 08:14:26.866351 30250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70799840b3490494f7e652fa41e133" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41397 } }
I20260620 08:14:26.866456 30250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:26.866503 30250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:26.866552 30250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:26.867139 30250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70799840b3490494f7e652fa41e133" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41397 } }
I20260620 08:14:26.867264 30250 leader_election.cc:302] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f70799840b3490494f7e652fa41e133; no voters:
I20260620 08:14:26.867462 30250 leader_election.cc:288] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:26.867516 30253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:26.867746 30250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:26.867798 30253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [term 1 LEADER]: Becoming Leader. State: Replica: 3f70799840b3490494f7e652fa41e133, State: Running, Role: LEADER
I20260620 08:14:26.867894 30253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70799840b3490494f7e652fa41e133" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41397 } }
I20260620 08:14:26.867941 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30183
I20260620 08:14:26.868036 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:26.868250 25574 external_mini_cluster.cc:1428] Setting key 56528477948c72b1e02eff20b0a1492b
I20260620 08:14:26.868552 30254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f70799840b3490494f7e652fa41e133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70799840b3490494f7e652fa41e133" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41397 } } }
I20260620 08:14:26.868635 30254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:26.868980 30255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f70799840b3490494f7e652fa41e133. Latest consensus state: current_term: 1 leader_uuid: "3f70799840b3490494f7e652fa41e133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70799840b3490494f7e652fa41e133" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41397 } } }
I20260620 08:14:26.869055 30255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133 [sys.catalog]: This master's current role is: LEADER
W20260620 08:14:26.869743 30270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:26.869801 30270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:26.869845 30263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:26.870066 30263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:26.871472 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:26.868941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.24.249.190:41397)
0620 08:14:26.869591 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:26.869600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:26.869629 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:14:26.869762 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:26.869769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:26.869821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:26.870139 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:26.870900 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:26.870903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:26.870904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:26.870908 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:26.870946 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:26.870960 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:26.871029 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:26.871113 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 08:14:26.871278 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":80,"threads_started":1}
I20260620 08:14:26.871815 30263 catalog_manager.cc:1347] Generated new cluster ID: b06401c47da541cf83f4312f515257b7
I20260620 08:14:26.871865 30263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:26.882128 30263 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:26.882318 30263 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:26.886011 30263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133: Generated new TSK 0
I20260620 08:14:26.886202 30263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:14:26.892349 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41397
--builtin_ntp_servers=127.24.249.148:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:14:26.978960 30274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:26.979213 30274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:26.979249 30274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:26.980480 30274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:26.980551 30274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:26.980592 30274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:26.982079 30274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:26.982350 30274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:26.982650 30274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:26.987735 30280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:26.987779 30279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.987905 30274 server_base.cc:1034] running on GCE node
W20260620 08:14:26.987970 30282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:26.988376 30274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:26.988716 30274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:26.989921 30274 hybrid_clock.cc:648] HybridClock initialized: now 1781943266989907 us; error 31 us; skew 500 ppm
I20260620 08:14:26.991412 30274 webserver.cc:460] Webserver started at http://127.24.249.129:36131/ using document root <none> and password file <none>
I20260620 08:14:26.991670 30274 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:26.991726 30274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:26.991855 30274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:26.993244 30274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "228a1ca1e02049b393d8be275ce67fc8"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "07c75e2391391da7b57ed25a18939ce7"
server_key_iv: "a563a23187c4e9bd6473d8c5fa7c7e7b"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.993629 30274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "228a1ca1e02049b393d8be275ce67fc8"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "07c75e2391391da7b57ed25a18939ce7"
server_key_iv: "a563a23187c4e9bd6473d8c5fa7c7e7b"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.995282 30274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 08:14:26.996173 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:26.996371 30274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:26.996481 30274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "228a1ca1e02049b393d8be275ce67fc8"
format_stamp: "Formatted at 2026-06-20 08:14:26 on dist-test-slave-65w8"
server_key: "07c75e2391391da7b57ed25a18939ce7"
server_key_iv: "a563a23187c4e9bd6473d8c5fa7c7e7b"
server_key_version: "encryptionkey@0"
I20260620 08:14:26.996574 30274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:27.001273 30274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:27.001516 30274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:27.001613 30274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:27.001765 30274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:27.002104 30274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:27.002148 30274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.002169 30274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:27.002223 30274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.007901 30274 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:36575
I20260620 08:14:27.007961 30401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:36575 every 8 connection(s)
I20260620 08:14:27.008296 30274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:27.011476 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.009188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:34241 (local address 127.24.249.190:41397)
0620 08:14:27.009259 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.009261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.009489 (+ 228us) server_negotiation.cc:407] Connection header received
0620 08:14:27.010021 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.010023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.010031 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.010084 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.011214 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.011231 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.011231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.011236 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.011251 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.011256 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.011306 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.011407 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.011421 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:14:27.011694 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.009120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41397 (local address 127.24.249.129:34241)
0620 08:14:27.009354 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.009388 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.009933 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.010147 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.010151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.010159 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.011059 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.011060 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.011088 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.011119 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.011326 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.011329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.011332 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.011391 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.011444 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260620 08:14:27.012365 30402 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41397
I20260620 08:14:27.012450 30402 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:27.012563 30402 heartbeater.cc:511] Master 127.24.249.190:41397 requested a full tablet report, sending...
I20260620 08:14:27.013059 30214 ts_manager.cc:194] Registered new tserver with Master: 228a1ca1e02049b393d8be275ce67fc8 (127.24.249.129:36575)
I20260620 08:14:27.017774 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30274
I20260620 08:14:27.017899 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:27.018054 25574 external_mini_cluster.cc:1428] Setting key 2ded7409bb13378d9f54f87032b9b6cd
I20260620 08:14:27.019001 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41397
--builtin_ntp_servers=127.24.249.148:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:14:27.100629 30405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:27.100839 30405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:27.100862 30405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:27.102165 30405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:27.102208 30405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:27.102234 30405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:27.103613 30405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:27.103881 30405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:27.104185 30405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:27.109444 30411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.109452 30413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.109413 30410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:27.109746 30405 server_base.cc:1034] running on GCE node
I20260620 08:14:27.110014 30405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:27.110236 30405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:27.111389 30405 hybrid_clock.cc:648] HybridClock initialized: now 1781943267111358 us; error 47 us; skew 500 ppm
I20260620 08:14:27.112773 30405 webserver.cc:460] Webserver started at http://127.24.249.130:39489/ using document root <none> and password file <none>
I20260620 08:14:27.113042 30405 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:27.113091 30405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:27.113289 30405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:27.114513 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "9f42b487b2fa4a4493cfbf640718acf6"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "8c3c1d2cb468904f40d4fad0ddddd209"
server_key_iv: "9612b51565942f218dcc130a284b3e15"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.114894 30405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "9f42b487b2fa4a4493cfbf640718acf6"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "8c3c1d2cb468904f40d4fad0ddddd209"
server_key_iv: "9612b51565942f218dcc130a284b3e15"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.116339 30405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:27.117123 30419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.117326 30405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:27.117429 30405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "9f42b487b2fa4a4493cfbf640718acf6"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "8c3c1d2cb468904f40d4fad0ddddd209"
server_key_iv: "9612b51565942f218dcc130a284b3e15"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.117516 30405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:27.120986 30405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:27.121260 30405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:27.121409 30405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:27.121619 30405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:27.121999 30405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:27.122051 30405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.122090 30405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:27.122117 30405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.128008 30405 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:32909
I20260620 08:14:27.128047 30532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:32909 every 8 connection(s)
I20260620 08:14:27.128384 30405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:27.131526 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.129100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:43713 (local address 127.24.249.190:41397)
0620 08:14:27.129145 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.129147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.129385 (+ 238us) server_negotiation.cc:407] Connection header received
0620 08:14:27.129961 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.129963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.129970 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.130033 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.131248 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.131259 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.131260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.131264 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.131280 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.131284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.131343 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.131440 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.131456 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:27.131731 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.129080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41397 (local address 127.24.249.130:43713)
0620 08:14:27.129262 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.129295 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.129801 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.130105 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.130114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.130122 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.131087 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.131087 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.131114 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.131142 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.131352 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.131357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.131360 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.131450 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.131511 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 08:14:27.132252 30533 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41397
I20260620 08:14:27.132328 30533 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:27.132431 30533 heartbeater.cc:511] Master 127.24.249.190:41397 requested a full tablet report, sending...
I20260620 08:14:27.132797 30214 ts_manager.cc:194] Registered new tserver with Master: 9f42b487b2fa4a4493cfbf640718acf6 (127.24.249.130:32909)
I20260620 08:14:27.134064 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30405
I20260620 08:14:27.134171 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:27.134378 25574 external_mini_cluster.cc:1428] Setting key a61637069e42ba656afed0faf7f7f823
I20260620 08:14:27.135293 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41397
--builtin_ntp_servers=127.24.249.148:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 08:14:27.219283 30536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:27.219493 30536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:27.219517 30536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:27.220819 30536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:27.220865 30536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:27.220891 30536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:27.222364 30536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:27.222611 30536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:27.222925 30536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:27.228116 30544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.228116 30542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:27.228366 30536 server_base.cc:1034] running on GCE node
W20260620 08:14:27.228564 30541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:27.228802 30536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:27.229066 30536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:27.230250 30536 hybrid_clock.cc:648] HybridClock initialized: now 1781943267230236 us; error 34 us; skew 500 ppm
I20260620 08:14:27.232120 30536 webserver.cc:460] Webserver started at http://127.24.249.131:41921/ using document root <none> and password file <none>
I20260620 08:14:27.232452 30536 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:27.232530 30536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:27.232699 30536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:27.234417 30536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "535b1dca12ff4e5abfd48c9a056089be"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "67cdb02932f8d5d50b7a3c205f31ed85"
server_key_iv: "91470b856daba9ff927c1478df5de8f1"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.234952 30536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "535b1dca12ff4e5abfd48c9a056089be"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "67cdb02932f8d5d50b7a3c205f31ed85"
server_key_iv: "91470b856daba9ff927c1478df5de8f1"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.236799 30536 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:27.237805 30550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.238023 30536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:27.238130 30536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "535b1dca12ff4e5abfd48c9a056089be"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "67cdb02932f8d5d50b7a3c205f31ed85"
server_key_iv: "91470b856daba9ff927c1478df5de8f1"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.238229 30536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:27.245065 30536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:27.245641 30536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:27.245810 30536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:27.246099 30536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:27.246512 30536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:27.246558 30536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.246593 30536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:27.246637 30536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.254557 30536 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:46617
I20260620 08:14:27.254594 30663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:46617 every 8 connection(s)
I20260620 08:14:27.255023 30536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:27.259058 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.255831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:60185 (local address 127.24.249.190:41397)
0620 08:14:27.256144 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.256147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.256262 (+ 115us) server_negotiation.cc:407] Connection header received
0620 08:14:27.257084 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.257085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.257091 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.257147 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.258673 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.258675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.258675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.258681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.258701 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.258705 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.258823 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.258962 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.258980 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260620 08:14:27.259500 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.255889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41397 (local address 127.24.249.131:60185)
0620 08:14:27.256128 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.256178 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.256959 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.257226 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.257230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.257243 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.258474 (+ 1231us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.258475 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.258519 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.258562 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.258854 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.258862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.258866 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.259181 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.259257 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260620 08:14:27.260005 30664 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41397
I20260620 08:14:27.260105 30664 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:27.260281 30664 heartbeater.cc:511] Master 127.24.249.190:41397 requested a full tablet report, sending...
I20260620 08:14:27.260738 30214 ts_manager.cc:194] Registered new tserver with Master: 535b1dca12ff4e5abfd48c9a056089be (127.24.249.131:46617)
I20260620 08:14:27.260763 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30536
I20260620 08:14:27.260869 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:27.261075 25574 external_mini_cluster.cc:1428] Setting key 4de79a0318d2ffff2150160a751bc7af
I20260620 08:14:27.262014 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:14:27.265066 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.263668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.24.249.190:41397)
0620 08:14:27.263722 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.263724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.263913 (+ 189us) server_negotiation.cc:407] Connection header received
0620 08:14:27.264014 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.264015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.264020 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.264062 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.264813 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.264814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.264814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.264818 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.264835 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.264840 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.264905 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.264972 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.264989 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:27.266943 30214 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:27.269150 30214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:27.273392 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.271349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:36575 (local address 127.0.0.1:46758)
0620 08:14:27.271470 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.271517 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.271578 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.272170 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.272181 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.272187 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.272967 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.272968 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.272986 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.273009 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.273247 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.273251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.273254 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.273310 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.273323 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260620 08:14:27.273494 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.271547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46617 (local address 127.0.0.1:50528)
0620 08:14:27.271673 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.271683 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.271720 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.272299 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.272300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.272303 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.273144 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.273145 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.273152 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.273166 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.273387 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.273388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.273389 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.273416 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.273429 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260620 08:14:27.273658 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.271444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.24.249.129:36575)
0620 08:14:27.271630 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.271635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.271646 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:27.271673 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.271675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.271685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.271762 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.273097 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.273104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.273104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.273109 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.273147 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.273176 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.273224 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.273562 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.273579 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260620 08:14:27.274003 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.271998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.24.249.131:46617)
0620 08:14:27.272144 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.272148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.272159 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:27.272183 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.272185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.272194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.272418 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.273248 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.273255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.273256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.273260 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.273297 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.273321 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.273654 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.273677 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.273692 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260620 08:14:27.274372 30336 tablet_service.cc:1467] Processing CreateTablet for tablet 79af8bb8915e44e3af7eca444f31d9d0 (DEFAULT_TABLE table=test-table [id=e1401fdd50404fc2abb1914e8dea0cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:27.274747 30336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af8bb8915e44e3af7eca444f31d9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:27.277829 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.271714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:32909 (local address 127.0.0.1:46794)
0620 08:14:27.271818 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.271828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.271862 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.272423 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.272424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.272427 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.277377 (+ 4950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.277378 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.277387 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.277403 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.277696 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.277697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.277698 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.277745 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.277759 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":72,"threads_started":1}
I20260620 08:14:27.278139 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.271807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.24.249.130:32909)
0620 08:14:27.271978 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.271982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.271992 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:27.272017 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.272019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.272029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.272099 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.277489 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.277501 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.277501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.277507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.277544 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.277578 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.277665 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.278023 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.278045 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260620 08:14:27.278384 30678 tablet_bootstrap.cc:492] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8: Bootstrap starting.
I20260620 08:14:27.278788 30467 tablet_service.cc:1467] Processing CreateTablet for tablet 79af8bb8915e44e3af7eca444f31d9d0 (DEFAULT_TABLE table=test-table [id=e1401fdd50404fc2abb1914e8dea0cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:27.279110 30467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af8bb8915e44e3af7eca444f31d9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:27.279215 30598 tablet_service.cc:1467] Processing CreateTablet for tablet 79af8bb8915e44e3af7eca444f31d9d0 (DEFAULT_TABLE table=test-table [id=e1401fdd50404fc2abb1914e8dea0cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:27.279517 30598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af8bb8915e44e3af7eca444f31d9d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:27.282406 30679 tablet_bootstrap.cc:492] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Bootstrap starting.
I20260620 08:14:27.282891 30678 tablet_bootstrap.cc:654] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:27.283186 30679 tablet_bootstrap.cc:654] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:27.283211 30678 log.cc:824] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:27.283528 30679 log.cc:824] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:27.283591 30680 tablet_bootstrap.cc:492] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Bootstrap starting.
I20260620 08:14:27.284049 30678 tablet_bootstrap.cc:492] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8: No bootstrap required, opened a new log
I20260620 08:14:27.284143 30678 ts_tablet_manager.cc:1397] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 08:14:27.284312 30679 tablet_bootstrap.cc:492] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: No bootstrap required, opened a new log
I20260620 08:14:27.284394 30679 ts_tablet_manager.cc:1397] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:27.286301 30678 raft_consensus.cc:348] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.286468 30678 raft_consensus.cc:374] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:27.286509 30678 raft_consensus.cc:729] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228a1ca1e02049b393d8be275ce67fc8, State: Initialized, Role: FOLLOWER
I20260620 08:14:27.286645 30678 consensus_queue.cc:260] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.286932 30678 ts_tablet_manager.cc:1428] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:14:27.287001 30402 heartbeater.cc:503] Master 127.24.249.190:41397 was elected leader, sending a full tablet report...
I20260620 08:14:27.287467 30679 raft_consensus.cc:348] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.287587 30680 tablet_bootstrap.cc:654] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:27.287624 30679 raft_consensus.cc:374] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:27.287659 30679 raft_consensus.cc:729] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f42b487b2fa4a4493cfbf640718acf6, State: Initialized, Role: FOLLOWER
I20260620 08:14:27.287850 30679 consensus_queue.cc:260] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.288144 30679 ts_tablet_manager.cc:1428] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:14:27.288190 30680 log.cc:824] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:27.288249 30533 heartbeater.cc:503] Master 127.24.249.190:41397 was elected leader, sending a full tablet report...
I20260620 08:14:27.289116 30680 tablet_bootstrap.cc:492] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: No bootstrap required, opened a new log
I20260620 08:14:27.289193 30680 ts_tablet_manager.cc:1397] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 08:14:27.292340 30680 raft_consensus.cc:348] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.292527 30680 raft_consensus.cc:374] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:27.292582 30680 raft_consensus.cc:729] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 535b1dca12ff4e5abfd48c9a056089be, State: Initialized, Role: FOLLOWER
I20260620 08:14:27.292708 30680 consensus_queue.cc:260] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.293030 30664 heartbeater.cc:503] Master 127.24.249.190:41397 was elected leader, sending a full tablet report...
I20260620 08:14:27.293015 30680 ts_tablet_manager.cc:1428] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:14:27.300062 30683 raft_consensus.cc:482] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:27.300177 30683 raft_consensus.cc:504] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.300590 30683 leader_election.cc:288] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 535b1dca12ff4e5abfd48c9a056089be (127.24.249.131:46617), 9f42b487b2fa4a4493cfbf640718acf6 (127.24.249.130:32909)
I20260620 08:14:27.301967 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.300769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:46617 (local address 127.24.249.129:55581)
0620 08:14:27.300830 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.300844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.300898 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.301076 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.301077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.301080 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.301606 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.301607 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.301615 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.301629 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.301815 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.301815 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.301816 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.301888 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.301904 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:14:27.302138 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.300839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:55581 (local address 127.24.249.131:46617)
0620 08:14:27.300905 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.300907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.300920 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:14:27.300949 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.300950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.300955 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.301016 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.301713 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.301715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.301715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.301719 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.301737 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.301742 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.301806 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.302059 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.302075 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:14:27.302145 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.301082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:56395 (local address 127.24.249.130:32909)
0620 08:14:27.301162 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.301164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.301173 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:27.301219 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.301220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.301224 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.301280 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.301931 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.301933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.301933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.301937 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.301954 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.301958 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.302016 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.302072 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.302086 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:14:27.302187 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.300875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:32909 (local address 127.24.249.129:56395)
0620 08:14:27.301069 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.301078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.301161 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.301285 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.301286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.301288 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:27.301811 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:27.301812 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:27.301818 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:27.301829 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:27.302014 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:27.302014 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:27.302015 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 08:14:27.302069 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 08:14:27.302085 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260620 08:14:27.302590 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af8bb8915e44e3af7eca444f31d9d0" candidate_uuid: "228a1ca1e02049b393d8be275ce67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "535b1dca12ff4e5abfd48c9a056089be" is_pre_election: true
I20260620 08:14:27.302614 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af8bb8915e44e3af7eca444f31d9d0" candidate_uuid: "228a1ca1e02049b393d8be275ce67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f42b487b2fa4a4493cfbf640718acf6" is_pre_election: true
I20260620 08:14:27.302744 30487 raft_consensus.cc:2454] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228a1ca1e02049b393d8be275ce67fc8 in term 0.
I20260620 08:14:27.302752 30618 raft_consensus.cc:2454] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228a1ca1e02049b393d8be275ce67fc8 in term 0.
I20260620 08:14:27.302955 30290 leader_election.cc:302] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228a1ca1e02049b393d8be275ce67fc8, 9f42b487b2fa4a4493cfbf640718acf6; no voters:
I20260620 08:14:27.303092 30683 raft_consensus.cc:2781] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:27.303179 30683 raft_consensus.cc:482] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:27.303215 30683 raft_consensus.cc:3037] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:27.303939 30683 raft_consensus.cc:504] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.304085 30683 leader_election.cc:288] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 535b1dca12ff4e5abfd48c9a056089be (127.24.249.131:46617), 9f42b487b2fa4a4493cfbf640718acf6 (127.24.249.130:32909)
I20260620 08:14:27.304314 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af8bb8915e44e3af7eca444f31d9d0" candidate_uuid: "228a1ca1e02049b393d8be275ce67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "535b1dca12ff4e5abfd48c9a056089be"
I20260620 08:14:27.304333 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af8bb8915e44e3af7eca444f31d9d0" candidate_uuid: "228a1ca1e02049b393d8be275ce67fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f42b487b2fa4a4493cfbf640718acf6"
I20260620 08:14:27.304414 30487 raft_consensus.cc:3037] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:27.304414 30618 raft_consensus.cc:3037] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:27.305351 30487 raft_consensus.cc:2454] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228a1ca1e02049b393d8be275ce67fc8 in term 1.
I20260620 08:14:27.305351 30618 raft_consensus.cc:2454] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228a1ca1e02049b393d8be275ce67fc8 in term 1.
I20260620 08:14:27.305584 30290 leader_election.cc:302] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228a1ca1e02049b393d8be275ce67fc8, 535b1dca12ff4e5abfd48c9a056089be; no voters:
I20260620 08:14:27.305841 30683 raft_consensus.cc:2781] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:27.306073 30683 raft_consensus.cc:686] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 228a1ca1e02049b393d8be275ce67fc8, State: Running, Role: LEADER
I20260620 08:14:27.306155 30683 consensus_queue.cc:237] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } }
I20260620 08:14:27.307065 30214 catalog_manager.cc:5495] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 228a1ca1e02049b393d8be275ce67fc8 (127.24.249.129). New cstate: current_term: 1 leader_uuid: "228a1ca1e02049b393d8be275ce67fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "228a1ca1e02049b393d8be275ce67fc8" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36575 } health_report { overall_health: HEALTHY } } }
I20260620 08:14:27.313963 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.312547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.24.249.129:36575)
0620 08:14:27.312631 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.312633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.312645 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:27.312735 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.312736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.312748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.312812 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.313590 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.313591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.313591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.313596 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.313611 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.313615 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.313678 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.313825 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.313839 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:14:27.315898 30618 raft_consensus.cc:1261] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 1 FOLLOWER]: Refusing update from remote peer 228a1ca1e02049b393d8be275ce67fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:27.315898 30487 raft_consensus.cc:1261] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 1 FOLLOWER]: Refusing update from remote peer 228a1ca1e02049b393d8be275ce67fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:27.316262 30688 consensus_queue.cc:1035] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535b1dca12ff4e5abfd48c9a056089be" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 46617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:27.316399 30683 consensus_queue.cc:1035] T 79af8bb8915e44e3af7eca444f31d9d0 P 228a1ca1e02049b393d8be275ce67fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f42b487b2fa4a4493cfbf640718acf6" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:27.319175 30691 mvcc.cc:205] Tried to move back new op lower bound from 7298839622922444800 to 7298839622886940672. Current Snapshot: MvccSnapshot[applied={T|T < 7298839622922444800}]
I20260620 08:14:27.320469 30692 mvcc.cc:205] Tried to move back new op lower bound from 7298839622922444800 to 7298839622886940672. Current Snapshot: MvccSnapshot[applied={T|T < 7298839622922444800}]
I20260620 08:14:27.321761 30214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:27.321903 30214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:27.322888 30214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f70799840b3490494f7e652fa41e133: Sending DeleteTablet for 3 replicas of tablet 79af8bb8915e44e3af7eca444f31d9d0
I20260620 08:14:27.323340 30336 tablet_service.cc:1514] Processing DeleteTablet for tablet 79af8bb8915e44e3af7eca444f31d9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:27 UTC) from {username='slave'} at 127.0.0.1:46758
I20260620 08:14:27.323446 30598 tablet_service.cc:1514] Processing DeleteTablet for tablet 79af8bb8915e44e3af7eca444f31d9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:27 UTC) from {username='slave'} at 127.0.0.1:50528
I20260620 08:14:27.323791 30701 tablet_replica.cc:330] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: stopping tablet replica
I20260620 08:14:27.323945 30701 raft_consensus.cc:2229] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:27.323975 30467 tablet_service.cc:1514] Processing DeleteTablet for tablet 79af8bb8915e44e3af7eca444f31d9d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:27 UTC) from {username='slave'} at 127.0.0.1:46794
I20260620 08:14:27.324106 30701 raft_consensus.cc:2258] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:27.324585 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30274
I20260620 08:14:27.325438 30701 ts_tablet_manager.cc:1905] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:27.327322 30701 ts_tablet_manager.cc:1918] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:27.327435 30701 log.cc:1196] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/79af8bb8915e44e3af7eca444f31d9d0
I20260620 08:14:27.327893 30701 ts_tablet_manager.cc:1939] T 79af8bb8915e44e3af7eca444f31d9d0 P 535b1dca12ff4e5abfd48c9a056089be: Deleting consensus metadata
I20260620 08:14:27.327960 30702 tablet_replica.cc:330] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: stopping tablet replica
I20260620 08:14:27.328107 30702 raft_consensus.cc:2229] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:27.328251 30702 raft_consensus.cc:2258] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:27.328516 30199 catalog_manager.cc:4841] TS 535b1dca12ff4e5abfd48c9a056089be (127.24.249.131:46617): tablet 79af8bb8915e44e3af7eca444f31d9d0 (table test-table [id=e1401fdd50404fc2abb1914e8dea0cb8]) successfully deleted
I20260620 08:14:27.328923 30702 ts_tablet_manager.cc:1905] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:27.330514 30702 ts_tablet_manager.cc:1918] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:27.330678 30702 log.cc:1196] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/79af8bb8915e44e3af7eca444f31d9d0
I20260620 08:14:27.331305 30702 ts_tablet_manager.cc:1939] T 79af8bb8915e44e3af7eca444f31d9d0 P 9f42b487b2fa4a4493cfbf640718acf6: Deleting consensus metadata
I20260620 08:14:27.331813 30199 catalog_manager.cc:4841] TS 9f42b487b2fa4a4493cfbf640718acf6 (127.24.249.130:32909): tablet 79af8bb8915e44e3af7eca444f31d9d0 (table test-table [id=e1401fdd50404fc2abb1914e8dea0cb8]) successfully deleted
W20260620 08:14:27.332268 30201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:36575 (error 108)
I20260620 08:14:27.332510 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30405
I20260620 08:14:27.333374 30675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.333241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:36575 (local address 127.0.0.1:46780)
0620 08:14:27.333291 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.333333 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:36575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 08:14:27.333503 30201 catalog_manager.cc:4584] TS 228a1ca1e02049b393d8be275ce67fc8 (127.24.249.129:36575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79af8bb8915e44e3af7eca444f31d9d0: Network error: Client connection negotiation failed: client connection to 127.24.249.129:36575: connect: Connection refused (error 111)
I20260620 08:14:27.337438 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30536
I20260620 08:14:27.341800 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30183
2026-06-20T08:14:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (617 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 08:14:27.362604 25574 mini_oidc.cc:152] Starting JWKS server
I20260620 08:14:27.363767 25574 webserver.cc:460] Webserver started at https://127.0.0.1:34717/ using document root <none> and password file <none>
I20260620 08:14:27.363916 25574 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:14:27.364108 25574 webserver.cc:460] Webserver started at http://127.0.0.1:44761/ using document root <none> and password file <none>
2026-06-20T08:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:27Z Disabled control of system clock
I20260620 08:14:27.372653 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:37483
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:37483
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34717/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/testchainca.cert with env {}
W20260620 08:14:27.453275 30709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:27.453488 30709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:27.453510 30709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:27.454722 30709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:27.454766 30709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:27.454778 30709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:27.454792 30709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:27.454803 30709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:14:27.454815 30709 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:14:27.454865 30709 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34717/jwks.json
I20260620 08:14:27.456419 30709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:37483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34717/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:27.456710 30709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:27.456949 30709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:27.461925 30717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.461936 30714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.461921 30715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:27.462163 30709 server_base.cc:1034] running on GCE node
I20260620 08:14:27.462394 30709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:27.462685 30709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:27.463850 30709 hybrid_clock.cc:648] HybridClock initialized: now 1781943267463825 us; error 38 us; skew 500 ppm
I20260620 08:14:27.465193 30709 webserver.cc:460] Webserver started at http://127.24.249.190:45687/ using document root <none> and password file <none>
I20260620 08:14:27.465449 30709 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:27.465497 30709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:27.465626 30709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:27.466796 30709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "7502af0bed6c4bff96a3a7fe35448d4b"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "f2647ff57c10cd27c152e3b68ecc36cb"
server_key_iv: "a3b742e462e23519dec4cd1ef8779b6b"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.467154 30709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "7502af0bed6c4bff96a3a7fe35448d4b"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "f2647ff57c10cd27c152e3b68ecc36cb"
server_key_iv: "a3b742e462e23519dec4cd1ef8779b6b"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.468662 30709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:27.469552 30723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.469798 30709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:27.469933 30709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "7502af0bed6c4bff96a3a7fe35448d4b"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "f2647ff57c10cd27c152e3b68ecc36cb"
server_key_iv: "a3b742e462e23519dec4cd1ef8779b6b"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.470002 30709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:27.479123 30709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:27.479391 30709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:27.479496 30709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:27.483547 30709 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:37483
I20260620 08:14:27.483595 30775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:37483 every 8 connection(s)
I20260620 08:14:27.483947 30709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:27.484954 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:27.487221 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30709
I20260620 08:14:27.487360 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:27.487560 25574 external_mini_cluster.cc:1428] Setting key d84e55df563ae70deb78c99ca4e61ce1
I20260620 08:14:27.487618 30776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b: Bootstrap starting.
I20260620 08:14:27.488329 30776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:27.488646 30776 log.cc:824] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:27.489336 30776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b: No bootstrap required, opened a new log
I20260620 08:14:27.491425 30776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37483 } }
I20260620 08:14:27.491565 30776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:27.491592 30776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7502af0bed6c4bff96a3a7fe35448d4b, State: Initialized, Role: FOLLOWER
I20260620 08:14:27.491755 30776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37483 } }
I20260620 08:14:27.491847 30776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:27.491879 30776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:27.491926 30776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:27.493024 30776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37483 } }
I20260620 08:14:27.493178 30776 leader_election.cc:302] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7502af0bed6c4bff96a3a7fe35448d4b; no voters:
I20260620 08:14:27.493343 30776 leader_election.cc:288] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:27.493414 30781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:27.493574 30781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [term 1 LEADER]: Becoming Leader. State: Replica: 7502af0bed6c4bff96a3a7fe35448d4b, State: Running, Role: LEADER
I20260620 08:14:27.493661 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.488091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.24.249.190:37483)
0620 08:14:27.488334 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.488341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.488389 (+ 48us) server_negotiation.cc:407] Connection header received
0620 08:14:27.488878 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.488885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.488947 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.489089 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.489342 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.490366 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.491169 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.491621 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.491781 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:27.492617 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.492620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.492621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.492860 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.492903 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.492919 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.492977 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.493188 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.493308 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260620 08:14:27.493685 30776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:27.493680 30781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37483 } }
I20260620 08:14:27.494143 30781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7502af0bed6c4bff96a3a7fe35448d4b. Latest consensus state: current_term: 1 leader_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37483 } } }
I20260620 08:14:27.494237 30781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:27.494374 30782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7502af0bed6c4bff96a3a7fe35448d4b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 37483 } } }
I20260620 08:14:27.494462 30782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:27.495657 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 08:14:27.495867 30796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:27.495921 30796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:27.495976 30787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:27.496140 30787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:27.497637 30787 catalog_manager.cc:1347] Generated new cluster ID: 0b9b3631e5f046fca6ec7f9e9411af24
I20260620 08:14:27.497690 30787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:27.507161 30787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:27.507889 30787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:27.517297 30787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7502af0bed6c4bff96a3a7fe35448d4b: Generated new TSK 0
I20260620 08:14:27.517529 30787 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 08:14:27.543337 30806 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 08:14:27.543622 30806 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 08:14:27.544049 30805 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:27.527336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:37483 (local address 127.0.0.1:54340)
0620 08:14:27.527567 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.527576 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.527637 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.527852 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.527853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.527855 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:14:27.527928 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:27.527930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.528902 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.528903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:27.529587 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:27.529590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.530149 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.530150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:27.530210 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:27.530234 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:14:27.543614 (+ 13380us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 08:14:27.543721 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.249.190:37483: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":112,"threads_started":1}
I20260620 08:14:27.544495 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.527407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.24.249.190:37483)
0620 08:14:27.527475 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.527478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.527642 (+ 164us) server_negotiation.cc:407] Connection header received
0620 08:14:27.527715 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.527716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.527730 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.527813 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:14:27.528011 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.528804 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.529683 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.530028 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.530102 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:27.530353 (+ 251us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:14:27.543423 (+ 13070us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 08:14:27.544303 (+ 880us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 08:14:27.544566 30779 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.249.190:37483: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 08:14:27.544668 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30709
2026-06-20T08:14:27Z chronyd exiting
I20260620 08:14:27.581199 25574 test_util.cc:175] -----------------------------------------------
I20260620 08:14:27.581318 25574 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781943243671421-25574-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 08:14:27.582224 25574 mini_oidc.cc:152] Starting JWKS server
I20260620 08:14:27.582923 25574 webserver.cc:460] Webserver started at https://127.0.0.1:38649/ using document root <none> and password file <none>
I20260620 08:14:27.582963 25574 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:14:27.583122 25574 webserver.cc:460] Webserver started at http://127.0.0.1:33205/ using document root <none> and password file <none>
2026-06-20T08:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:27Z Disabled control of system clock
I20260620 08:14:27.591507 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:36463
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:36463
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38649/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/testchainca.cert with env {}
W20260620 08:14:27.673906 30812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:27.674121 30812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:27.674151 30812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:27.675297 30812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:27.675369 30812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:27.675395 30812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:27.675416 30812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:27.675448 30812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:14:27.675483 30812 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:14:27.675496 30812 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38649/jwks.json
I20260620 08:14:27.676992 30812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:36463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38649/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:27.677188 30812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:27.677350 30812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:27.682857 30818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.682874 30817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.682865 30820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:27.683233 30812 server_base.cc:1034] running on GCE node
I20260620 08:14:27.683481 30812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:27.683786 30812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:27.684958 30812 hybrid_clock.cc:648] HybridClock initialized: now 1781943267684931 us; error 40 us; skew 500 ppm
I20260620 08:14:27.686422 30812 webserver.cc:460] Webserver started at http://127.24.249.190:41907/ using document root <none> and password file <none>
I20260620 08:14:27.686635 30812 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:27.686676 30812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:27.686821 30812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:27.687836 30812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "cf3568e8e11d4ca9890227349913cccd"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "6a725512381311896f40ed865a98cc6e"
server_key_iv: "7400ee61d0004bd571f810086fea3428"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.688123 30812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "cf3568e8e11d4ca9890227349913cccd"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "6a725512381311896f40ed865a98cc6e"
server_key_iv: "7400ee61d0004bd571f810086fea3428"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.689363 30812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:27.690264 30826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.690512 30812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:27.690608 30812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "cf3568e8e11d4ca9890227349913cccd"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "6a725512381311896f40ed865a98cc6e"
server_key_iv: "7400ee61d0004bd571f810086fea3428"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.690687 30812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:27.714298 30812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:27.714603 30812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:27.714716 30812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:27.718644 30812 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:36463
I20260620 08:14:27.718678 30878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:36463 every 8 connection(s)
I20260620 08:14:27.719064 30812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:27.719741 30879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:27.722667 30879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd: Bootstrap starting.
I20260620 08:14:27.723248 30879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:27.723577 30879 log.cc:824] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:27.724337 30879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd: No bootstrap required, opened a new log
I20260620 08:14:27.726013 30879 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3568e8e11d4ca9890227349913cccd" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36463 } }
I20260620 08:14:27.726200 30879 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:27.726239 30879 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3568e8e11d4ca9890227349913cccd, State: Initialized, Role: FOLLOWER
I20260620 08:14:27.726298 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30812
I20260620 08:14:27.726456 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:27.726435 30879 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3568e8e11d4ca9890227349913cccd" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36463 } }
I20260620 08:14:27.726523 30879 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:27.726565 30879 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:27.726616 30879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:27.726678 25574 external_mini_cluster.cc:1428] Setting key 40587f3812393ba3456ac7ac70b2e644
I20260620 08:14:27.727454 30879 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3568e8e11d4ca9890227349913cccd" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36463 } }
I20260620 08:14:27.727566 30879 leader_election.cc:302] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf3568e8e11d4ca9890227349913cccd; no voters:
I20260620 08:14:27.727742 30879 leader_election.cc:288] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:27.727777 30884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:27.727888 30884 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [term 1 LEADER]: Becoming Leader. State: Replica: cf3568e8e11d4ca9890227349913cccd, State: Running, Role: LEADER
I20260620 08:14:27.727967 30884 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3568e8e11d4ca9890227349913cccd" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36463 } }
I20260620 08:14:27.728000 30879 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:27.728440 30885 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf3568e8e11d4ca9890227349913cccd. Latest consensus state: current_term: 1 leader_uuid: "cf3568e8e11d4ca9890227349913cccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3568e8e11d4ca9890227349913cccd" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36463 } } }
I20260620 08:14:27.728516 30885 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:27.728704 30889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:27.729234 30889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:27.729821 30886 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf3568e8e11d4ca9890227349913cccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3568e8e11d4ca9890227349913cccd" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36463 } } }
I20260620 08:14:27.729985 30886 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:27.733498 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.727314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.24.249.190:36463)
0620 08:14:27.727526 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.727533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.727581 (+ 48us) server_negotiation.cc:407] Connection header received
0620 08:14:27.727749 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.727754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.727844 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.728278 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.728390 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.730333 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.731333 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.731756 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.731891 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:27.732786 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.732791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.732791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.732982 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.733017 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.733029 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.733107 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.733218 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.733307 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260620 08:14:27.734639 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:14:27.739099 30889 catalog_manager.cc:1347] Generated new cluster ID: 593a463c31af454b9607639782170ea8
I20260620 08:14:27.739218 30889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:27.746734 30889 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:27.747457 30889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:27.757252 30889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf3568e8e11d4ca9890227349913cccd: Generated new TSK 0
I20260620 08:14:27.757477 30889 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 08:14:27.784554 30909 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 08:14:27.784644 30909 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 08:14:27.784991 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.762470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.24.249.190:36463)
0620 08:14:27.762535 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.762537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.762654 (+ 117us) server_negotiation.cc:407] Connection header received
0620 08:14:27.762725 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.762726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.762737 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.762797 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:14:27.762961 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.763471 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.764635 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.764856 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.764921 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:27.765147 (+ 226us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:14:27.784774 (+ 19627us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 08:14:27.784935 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 08:14:27.785064 30883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 08:14:27.785140 30908 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:27.762416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:36463 (local address 127.0.0.1:33802)
0620 08:14:27.762574 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:27.762586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:27.762654 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:27.762822 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:27.762823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:27.762826 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:14:27.762891 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:27.762893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.763567 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.763568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:27.764522 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:27.764525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.764947 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.764948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:27.765011 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:27.765016 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:14:27.784913 (+ 19897us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 08:14:27.785044 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.249.190:36463: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38649/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260620 08:14:27.785452 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30812
2026-06-20T08:14:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 08:14:27.823760 25574 mini_oidc.cc:152] Starting JWKS server
I20260620 08:14:27.824522 25574 webserver.cc:460] Webserver started at https://127.0.0.1:36861/ using document root <none> and password file <none>
I20260620 08:14:27.824595 25574 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 08:14:27.824797 25574 webserver.cc:460] Webserver started at http://127.0.0.1:40767/ using document root <none> and password file <none>
2026-06-20T08:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:27Z Disabled control of system clock
I20260620 08:14:27.833320 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:44763
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:44763
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 08:14:27.917891 30915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:27.918167 30915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:27.918196 30915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:27.919598 30915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:27.919647 30915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:27.919660 30915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:27.919673 30915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:27.919685 30915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:14:27.919698 30915 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:14:27.919742 30915 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36861/jwks.json
I20260620 08:14:27.922004 30915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:44763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:44763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:27.922322 30915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:27.922665 30915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:27.927883 30921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.927883 30923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:27.927914 30920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:27.928272 30915 server_base.cc:1034] running on GCE node
I20260620 08:14:27.928537 30915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:27.928839 30915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:27.930008 30915 hybrid_clock.cc:648] HybridClock initialized: now 1781943267929993 us; error 33 us; skew 500 ppm
I20260620 08:14:27.931571 30915 webserver.cc:460] Webserver started at http://127.24.249.190:45229/ using document root <none> and password file <none>
I20260620 08:14:27.931838 30915 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:27.931913 30915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:27.932062 30915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:27.933346 30915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "5fdd9e7404fa44a5a4e06516f223ec26"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "41d732f3342e45a9ee103d49ff62cf97"
server_key_iv: "0304bd60b97b651d8d15ebae15ed4c88"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.933786 30915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "5fdd9e7404fa44a5a4e06516f223ec26"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "41d732f3342e45a9ee103d49ff62cf97"
server_key_iv: "0304bd60b97b651d8d15ebae15ed4c88"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.935313 30915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:27.936141 30929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:27.936390 30915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:27.936506 30915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "5fdd9e7404fa44a5a4e06516f223ec26"
format_stamp: "Formatted at 2026-06-20 08:14:27 on dist-test-slave-65w8"
server_key: "41d732f3342e45a9ee103d49ff62cf97"
server_key_iv: "0304bd60b97b651d8d15ebae15ed4c88"
server_key_version: "encryptionkey@0"
I20260620 08:14:27.936583 30915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:27.961781 30915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:27.962153 30915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:27.962321 30915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:27.967121 30915 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:44763
I20260620 08:14:27.967164 30981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:44763 every 8 connection(s)
I20260620 08:14:27.967547 30915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:27.968446 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 30915
I20260620 08:14:27.968446 30982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:27.968567 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:27.968813 25574 external_mini_cluster.cc:1428] Setting key 6bfd18d91e046f83c43a1763d548e5bd
I20260620 08:14:27.973454 30982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26: Bootstrap starting.
I20260620 08:14:27.974213 30982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:27.974582 30982 log.cc:824] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:27.975502 30982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26: No bootstrap required, opened a new log
I20260620 08:14:27.975992 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:27.969410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.24.249.190:44763)
0620 08:14:27.969656 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:27.969667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:27.969704 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:14:27.970948 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:27.970957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:27.971034 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:27.971166 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:27.971489 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.972453 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.973738 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:27.974249 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:27.974387 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:27.975239 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:27.975243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:27.975243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:27.975407 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:27.975438 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:27.975449 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:27.975502 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:27.975659 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 08:14:27.975819 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260620 08:14:27.977202 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:14:27.977923 30982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44763 } }
I20260620 08:14:27.978128 30982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:27.978183 30982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdd9e7404fa44a5a4e06516f223ec26, State: Initialized, Role: FOLLOWER
I20260620 08:14:27.978320 30982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44763 } }
I20260620 08:14:27.978420 30982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:27.978468 30982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:27.978523 30982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:28.018710 30982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44763 } }
I20260620 08:14:28.018990 30982 leader_election.cc:302] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fdd9e7404fa44a5a4e06516f223ec26; no voters:
I20260620 08:14:28.019256 30982 leader_election.cc:288] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:28.019316 30987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:28.019481 30987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [term 1 LEADER]: Becoming Leader. State: Replica: 5fdd9e7404fa44a5a4e06516f223ec26, State: Running, Role: LEADER
I20260620 08:14:28.019574 30982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:28.019598 30987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44763 } }
I20260620 08:14:28.019948 30988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44763 } } }
I20260620 08:14:28.020054 30988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:28.020218 30989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fdd9e7404fa44a5a4e06516f223ec26. Latest consensus state: current_term: 1 leader_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fdd9e7404fa44a5a4e06516f223ec26" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44763 } } }
I20260620 08:14:28.020295 30989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:28.020360 30993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:28.020908 30993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:28.022783 30993 catalog_manager.cc:1347] Generated new cluster ID: 55a2962f931d4c20b1c457ec62270d66
I20260620 08:14:28.022835 30993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:28.058424 30993 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:28.059000 30993 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:28.075138 30993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fdd9e7404fa44a5a4e06516f223ec26: Generated new TSK 0
I20260620 08:14:28.075371 30993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:14:28.096801 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:28.092500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.24.249.190:44763)
0620 08:14:28.092566 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:28.092568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:28.092659 (+ 91us) server_negotiation.cc:407] Connection header received
0620 08:14:28.092785 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:28.092786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:28.092801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:28.092874 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:14:28.093087 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.093816 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.094834 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.095110 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.095191 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:28.095405 (+ 214us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:14:28.096554 (+ 1149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 08:14:28.096720 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 08:14:28.096879 30985 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 08:14:28.098747 31011 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:28.092433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:44763 (local address 127.0.0.1:53512)
0620 08:14:28.092578 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:28.092588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:28.092657 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:28.092915 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:28.092916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:28.092919 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:14:28.092975 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:28.092977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.093970 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.093971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:28.094713 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:28.094743 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.095219 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.095220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:28.095277 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:28.095283 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:14:28.096682 (+ 1399us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 08:14:28.098639 (+ 1957us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.249.190:44763: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1738,"thread_start_us":98,"threads_started":1}
I20260620 08:14:28.099036 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 30915
E20260620 08:14:28.110338 31012 webserver.cc:533] Webserver: sslize failed:
E20260620 08:14:28.110435 31012 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T08:14:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T08:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:28Z Disabled control of system clock
I20260620 08:14:28.147079 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:33985
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:33985
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/testchainca.cert with env {}
W20260620 08:14:28.233297 31016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:28.233534 31016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:28.233559 31016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:28.234824 31016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:28.234871 31016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:28.234884 31016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:28.234896 31016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:28.234910 31016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 08:14:28.234922 31016 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 08:14:28.234933 31016 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 08:14:28.236469 31016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:33985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:28.236688 31016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:28.236912 31016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:28.242182 31024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:28.242213 31021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:28.242404 31022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:28.242553 31016 server_base.cc:1034] running on GCE node
I20260620 08:14:28.242815 31016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:28.243153 31016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:28.244354 31016 hybrid_clock.cc:648] HybridClock initialized: now 1781943268244313 us; error 59 us; skew 500 ppm
I20260620 08:14:28.245920 31016 webserver.cc:460] Webserver started at http://127.24.249.190:36627/ using document root <none> and password file <none>
I20260620 08:14:28.246189 31016 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:28.246241 31016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:28.246381 31016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:28.247588 31016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "09e3c67f82be429896c81741ffcd3856"
format_stamp: "Formatted at 2026-06-20 08:14:28 on dist-test-slave-65w8"
server_key: "1442564583aa59d0c5a858a5487a87f8"
server_key_iv: "d8a8814ab4589477ba941657f70e3aea"
server_key_version: "encryptionkey@0"
I20260620 08:14:28.247967 31016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "09e3c67f82be429896c81741ffcd3856"
format_stamp: "Formatted at 2026-06-20 08:14:28 on dist-test-slave-65w8"
server_key: "1442564583aa59d0c5a858a5487a87f8"
server_key_iv: "d8a8814ab4589477ba941657f70e3aea"
server_key_version: "encryptionkey@0"
I20260620 08:14:28.249573 31016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:28.250560 31030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:28.250803 31016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:28.250939 31016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "09e3c67f82be429896c81741ffcd3856"
format_stamp: "Formatted at 2026-06-20 08:14:28 on dist-test-slave-65w8"
server_key: "1442564583aa59d0c5a858a5487a87f8"
server_key_iv: "d8a8814ab4589477ba941657f70e3aea"
server_key_version: "encryptionkey@0"
I20260620 08:14:28.251031 31016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:28.274004 31016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:28.274382 31016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:28.274541 31016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:28.278492 31016 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:33985
I20260620 08:14:28.278568 31082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:33985 every 8 connection(s)
I20260620 08:14:28.278887 31016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:28.280082 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:28.282294 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31016
I20260620 08:14:28.282409 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:28.282610 25574 external_mini_cluster.cc:1428] Setting key 3e687c6fa98073faef82728f6250add2
I20260620 08:14:28.283033 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856: Bootstrap starting.
I20260620 08:14:28.283700 31083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:28.284077 31083 log.cc:824] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:28.284825 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856: No bootstrap required, opened a new log
I20260620 08:14:28.286893 31083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e3c67f82be429896c81741ffcd3856" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33985 } }
I20260620 08:14:28.287030 31083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:28.287053 31083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e3c67f82be429896c81741ffcd3856, State: Initialized, Role: FOLLOWER
I20260620 08:14:28.287215 31083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e3c67f82be429896c81741ffcd3856" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33985 } }
I20260620 08:14:28.287305 31083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:28.287333 31083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:28.287403 31083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:28.289100 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:28.283932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.24.249.190:33985)
0620 08:14:28.284308 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:28.284316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:28.284342 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:14:28.284790 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:28.284796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:28.284883 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:28.285054 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:28.285274 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.286231 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.287012 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.287638 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.287781 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:28.288462 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:28.288466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:28.288466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:28.288652 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:28.288690 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:28.288704 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:28.288788 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:28.288867 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:14:28.288948 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":102,"threads_started":1}
I20260620 08:14:28.290437 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:14:28.320266 31083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e3c67f82be429896c81741ffcd3856" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33985 } }
I20260620 08:14:28.320434 31083 leader_election.cc:302] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e3c67f82be429896c81741ffcd3856; no voters:
I20260620 08:14:28.320685 31083 leader_election.cc:288] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:28.320780 31088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:28.320963 31088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [term 1 LEADER]: Becoming Leader. State: Replica: 09e3c67f82be429896c81741ffcd3856, State: Running, Role: LEADER
I20260620 08:14:28.321031 31083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:28.321074 31088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e3c67f82be429896c81741ffcd3856" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33985 } }
I20260620 08:14:28.321535 31089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09e3c67f82be429896c81741ffcd3856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e3c67f82be429896c81741ffcd3856" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33985 } } }
I20260620 08:14:28.321640 31089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:28.321841 31090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09e3c67f82be429896c81741ffcd3856. Latest consensus state: current_term: 1 leader_uuid: "09e3c67f82be429896c81741ffcd3856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e3c67f82be429896c81741ffcd3856" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33985 } } }
I20260620 08:14:28.321990 31090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:28.322184 31097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:28.322789 31097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:28.324299 31097 catalog_manager.cc:1347] Generated new cluster ID: 7f159d2e192f4585a56bf2519413fca5
I20260620 08:14:28.324368 31097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:28.336308 31097 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:28.337057 31097 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:28.339901 31097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09e3c67f82be429896c81741ffcd3856: Generated new TSK 0
I20260620 08:14:28.340085 31097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:14:28.439276 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:28.345987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.24.249.190:33985)
0620 08:14:28.346053 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:28.346056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:28.346065 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:28.346101 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:28.346102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:28.346114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:28.346318 (+ 204us) server_negotiation.cc:226] Negotiated authn=JWT
0620 08:14:28.346460 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.347338 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.348142 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.348359 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.348447 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:28.348702 (+ 255us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 08:14:28.439055 (+ 90353us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 08:14:28.439209 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54640: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 08:14:28.439368 31086 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54640: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 08:14:28.439513 31112 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:28.345798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:33985 (local address 127.0.0.1:54640)
0620 08:14:28.345942 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:28.345953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:28.346032 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:28.346318 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:28.346319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:28.346322 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 08:14:28.346385 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:28.346388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.347442 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.347443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:28.348052 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:28.348053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:28.348457 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:28.348458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:28.348517 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:28.348522 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 08:14:28.439213 (+ 90691us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 08:14:28.439400 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.249.190:33985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":96,"threads_started":1}
I20260620 08:14:28.439869 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31016
2026-06-20T08:14:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (319 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (103 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:30 dist-test-slave-65w8 krb5kdc[31124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:30 dist-test-slave-65w8 krb5kdc[31124](info): set up 2 sockets
Jun 20 08:14:30 dist-test-slave-65w8 krb5kdc[31124](info): commencing operation
krb5kdc: starting...
W20260620 08:14:31.306391 25574 mini_kdc.cc:121] Time spent starting KDC: real 2.655s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:31 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:31.437819 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40551
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:40551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:31.526222 31137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:31.526443 31137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:31.526466 31137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:31.527704 31137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:31.527753 31137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:31.527772 31137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:31.527786 31137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:31.527798 31137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:31.529377 31137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:40551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:31.529647 31137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:31.529934 31137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:31.535254 31142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:31.535254 31145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:31.535436 31143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:31.536206 31137 server_base.cc:1034] running on GCE node
I20260620 08:14:31.536563 31137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:31.536948 31137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:31.538162 31137 hybrid_clock.cc:648] HybridClock initialized: now 1781943271538137 us; error 41 us; skew 500 ppm
Jun 20 08:14:31 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943271, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:31.541043 31137 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:31.541827 31137 webserver.cc:460] Webserver started at http://127.24.249.190:41779/ using document root <none> and password file <none>
I20260620 08:14:31.542120 31137 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:31.542196 31137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:31.542357 31137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:31.543509 31137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "0c3f5cf699e547529f6b3c74f83e1343"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "cb838371c33c9d8588a78c32613dffcf"
server_key_iv: "09ed63ff24a9132685865040abc6b3ec"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.543910 31137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "0c3f5cf699e547529f6b3c74f83e1343"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "cb838371c33c9d8588a78c32613dffcf"
server_key_iv: "09ed63ff24a9132685865040abc6b3ec"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.545342 31137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:31.546420 31152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:31.546636 31137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:31.546754 31137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "0c3f5cf699e547529f6b3c74f83e1343"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "cb838371c33c9d8588a78c32613dffcf"
server_key_iv: "09ed63ff24a9132685865040abc6b3ec"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.546873 31137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:31.562664 31137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:31.565306 31137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:31.565505 31137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:31.570662 31137 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:40551
I20260620 08:14:31.570750 31204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:40551 every 8 connection(s)
I20260620 08:14:31.571146 31137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:31.572085 31205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:31.573529 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31137
I20260620 08:14:31.573660 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:31.574208 25574 external_mini_cluster.cc:1428] Setting key e1a9a95be916b7afa28da6184b17d5e5
I20260620 08:14:31.574707 31205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343: Bootstrap starting.
I20260620 08:14:31.575379 31205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:31.575686 31205 log.cc:824] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:31.576399 31205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343: No bootstrap required, opened a new log
I20260620 08:14:31.578306 31205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3f5cf699e547529f6b3c74f83e1343" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40551 } }
I20260620 08:14:31.578473 31205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:31.578521 31205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3f5cf699e547529f6b3c74f83e1343, State: Initialized, Role: FOLLOWER
I20260620 08:14:31.578675 31205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3f5cf699e547529f6b3c74f83e1343" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40551 } }
I20260620 08:14:31.578765 31205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:31.578794 31205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:31.578831 31205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:31.579571 31205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3f5cf699e547529f6b3c74f83e1343" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40551 } }
I20260620 08:14:31.579824 31205 leader_election.cc:302] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3f5cf699e547529f6b3c74f83e1343; no voters:
I20260620 08:14:31.579996 31205 leader_election.cc:288] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:31.580078 31210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:31.580215 31210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [term 1 LEADER]: Becoming Leader. State: Replica: 0c3f5cf699e547529f6b3c74f83e1343, State: Running, Role: LEADER
I20260620 08:14:31.580308 31205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:31.580319 31210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3f5cf699e547529f6b3c74f83e1343" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40551 } }
I20260620 08:14:31.580811 31210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c3f5cf699e547529f6b3c74f83e1343. Latest consensus state: current_term: 1 leader_uuid: "0c3f5cf699e547529f6b3c74f83e1343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3f5cf699e547529f6b3c74f83e1343" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40551 } } }
I20260620 08:14:31.580883 31210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:31.581068 31211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c3f5cf699e547529f6b3c74f83e1343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3f5cf699e547529f6b3c74f83e1343" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40551 } } }
I20260620 08:14:31.581147 31211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:31.581358 31214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:31.581921 31214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:14:31 dist-test-slave-65w8 krb5kdc[31124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:31.583734 31214 catalog_manager.cc:1347] Generated new cluster ID: 8f33ca90dc9a4246b38f1e76a1b5ae03
I20260620 08:14:31.583793 31214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:31.587159 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:31.574879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.24.249.190:40551)
0620 08:14:31.575053 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:31.575061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:31.575085 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:14:31.575592 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:31.575600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:31.575657 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:31.575819 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:31.576566 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.577613 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.578621 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.579232 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.579356 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:31.583493 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:31.583497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:31.583498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:31.583512 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:31.585506 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:31.585955 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:31.585959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:31.585959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:31.586045 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:31.586267 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:31.586268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:31.586269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:31.586510 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:31.586723 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:31.586824 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 08:14:31.586973 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260620 08:14:31.608763 31214 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:31.609349 31214 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:31.616475 31214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c3f5cf699e547529f6b3c74f83e1343: Generated new TSK 0
I20260620 08:14:31.616717 31214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:31.648538 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40551
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:31.744944 31231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:31.745256 31231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:31.745304 31231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:31.746955 31231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:31.747027 31231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:31.747076 31231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:31.748709 31231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:31.749004 31231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:31.749265 31231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:31.754658 31237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:31.754675 31236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:31.754675 31239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:31.754895 31231 server_base.cc:1034] running on GCE node
I20260620 08:14:31.755208 31231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:31.755492 31231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:31.756667 31231 hybrid_clock.cc:648] HybridClock initialized: now 1781943271756640 us; error 44 us; skew 500 ppm
Jun 20 08:14:31 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943271, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:31.759248 31231 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:31.759697 31231 webserver.cc:460] Webserver started at http://127.24.249.129:42895/ using document root <none> and password file <none>
I20260620 08:14:31.759948 31231 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:31.760012 31231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:31.760102 31231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:31.761171 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "d5544e0aaf6343c687a181cb95caa52f"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "b46088a41c68a44374197ce1bcddd990"
server_key_iv: "cf4e57650b17a0a3b97005b768d56924"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.761475 31231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "d5544e0aaf6343c687a181cb95caa52f"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "b46088a41c68a44374197ce1bcddd990"
server_key_iv: "cf4e57650b17a0a3b97005b768d56924"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.762879 31231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:31.763604 31246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:31.763867 31231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:31.763955 31231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "d5544e0aaf6343c687a181cb95caa52f"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "b46088a41c68a44374197ce1bcddd990"
server_key_iv: "cf4e57650b17a0a3b97005b768d56924"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.764010 31231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:31.775144 31231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:31.777145 31231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:31.777261 31231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:31.777446 31231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:31.777777 31231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:31.777814 31231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:31.777837 31231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:31.777875 31231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:31.783784 31231 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:36927
I20260620 08:14:31.783842 31359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:36927 every 8 connection(s)
I20260620 08:14:31.784164 31231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:31.785152 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31231
I20260620 08:14:31.785295 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:31.785496 25574 external_mini_cluster.cc:1428] Setting key 9e4aa28e36428e695e3356cb96f7f3ba
Jun 20 08:14:31 dist-test-slave-65w8 krb5kdc[31124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943271, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:14:31.795214 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:31.784998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:48681 (local address 127.24.249.190:40551)
0620 08:14:31.785049 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:31.785052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:31.785275 (+ 223us) server_negotiation.cc:407] Connection header received
0620 08:14:31.786541 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:31.786542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:31.786555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:31.786601 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:31.787258 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.788009 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.789077 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.789419 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.789506 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:31.792064 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:31.792065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:31.792066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:31.792074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:31.793787 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:31.794268 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:31.794270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:31.794270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:31.794335 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:31.794579 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:31.794580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:31.794580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:31.794746 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:31.794817 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:31.795068 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 08:14:31.795149 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:14:31.795356 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:31.784888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40551 (local address 127.24.249.129:48681)
0620 08:14:31.785059 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:31.785129 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:31.786439 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:31.786670 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:31.786674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:31.786959 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:31.787152 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:31.787156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.788129 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.788130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:31.788988 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:31.788990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.789508 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.789508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:31.789650 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:31.790269 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:31.790278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:31.791966 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:31.793950 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:31.793957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:31.793959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:31.794188 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:31.794407 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:31.794407 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:31.794407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:31.794502 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:31.794812 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:31.794818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:31.794934 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 08:14:31.795016 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 08:14:31.795138 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260620 08:14:31.795931 31360 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40551
I20260620 08:14:31.796037 31360 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:31.796211 31360 heartbeater.cc:511] Master 127.24.249.190:40551 requested a full tablet report, sending...
I20260620 08:14:31.796697 31169 ts_manager.cc:194] Registered new tserver with Master: d5544e0aaf6343c687a181cb95caa52f (127.24.249.129:36927)
I20260620 08:14:31.797509 31169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:48681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:31.813309 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40551
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:31.899924 31365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:31.900167 31365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:31.900204 31365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:31.901580 31365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:31.901682 31365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:31.901736 31365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:31.903384 31365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:31.903697 31365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:31.903946 31365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:31.909219 31373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:31.909219 31371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:31.909308 31370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:31.909909 31365 server_base.cc:1034] running on GCE node
I20260620 08:14:31.910159 31365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:31.910431 31365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:31.911641 31365 hybrid_clock.cc:648] HybridClock initialized: now 1781943271911624 us; error 28 us; skew 500 ppm
Jun 20 08:14:31 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943271, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:31.914400 31365 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:31.914927 31365 webserver.cc:460] Webserver started at http://127.24.249.130:40925/ using document root <none> and password file <none>
I20260620 08:14:31.915174 31365 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:31.915241 31365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:31.915395 31365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:31.916482 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "cd3f6a68fdc94bb4a36d63813c03a1fb"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "d77cea4e3df42ac09a8f11239baf8f4a"
server_key_iv: "6b48596e0c19711d1d3f4f07273a8a4e"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.916857 31365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "cd3f6a68fdc94bb4a36d63813c03a1fb"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "d77cea4e3df42ac09a8f11239baf8f4a"
server_key_iv: "6b48596e0c19711d1d3f4f07273a8a4e"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.918223 31365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:31.919080 31380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:31.919262 31365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:31.919366 31365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "cd3f6a68fdc94bb4a36d63813c03a1fb"
format_stamp: "Formatted at 2026-06-20 08:14:31 on dist-test-slave-65w8"
server_key: "d77cea4e3df42ac09a8f11239baf8f4a"
server_key_iv: "6b48596e0c19711d1d3f4f07273a8a4e"
server_key_version: "encryptionkey@0"
I20260620 08:14:31.919440 31365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:31.939237 31365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:31.941558 31365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:31.941748 31365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:31.942093 31365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:31.942440 31365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:31.942497 31365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:31.942539 31365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:31.942567 31365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:31.948469 31365 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:44293
I20260620 08:14:31.948511 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:44293 every 8 connection(s)
I20260620 08:14:31.948832 31365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:31.949077 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31365
I20260620 08:14:31.949198 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:31.949414 25574 external_mini_cluster.cc:1428] Setting key fd56c06417de00eab0a53b09b185a560
Jun 20 08:14:31 dist-test-slave-65w8 krb5kdc[31124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943271, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:14:31.960143 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:31.949714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:55947 (local address 127.24.249.190:40551)
0620 08:14:31.949765 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:31.949767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:31.950048 (+ 281us) server_negotiation.cc:407] Connection header received
0620 08:14:31.951116 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:31.951117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:31.951130 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:31.951197 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:31.951930 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.952460 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.954108 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.954398 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.954464 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:31.957080 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:31.957082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:31.957082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:31.957090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:31.958653 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:31.959124 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:31.959126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:31.959126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:31.959186 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:31.959464 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:31.959465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:31.959465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:31.959594 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:31.959676 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:31.959955 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 08:14:31.960047 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:14:31.960335 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:31.949703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40551 (local address 127.24.249.130:55947)
0620 08:14:31.949921 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:31.949962 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:31.951004 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:31.951255 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:31.951258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:31.951525 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:31.951832 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:31.951838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.952557 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.952558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:31.953996 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:31.953999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:31.954493 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:31.954494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:31.954658 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:31.955180 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:31.955190 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:31.956966 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:31.958789 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:31.958796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:31.958798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:31.959026 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:31.959272 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:31.959273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:31.959273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:31.959391 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:31.959666 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:31.959672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:31.959820 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 08:14:31.959933 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 08:14:31.960095 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":118,"threads_started":1}
I20260620 08:14:31.961056 31494 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40551
I20260620 08:14:31.961167 31494 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:31.961338 31494 heartbeater.cc:511] Master 127.24.249.190:40551 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
I20260620 08:14:31.961796 31169 ts_manager.cc:194] Registered new tserver with Master: cd3f6a68fdc94bb4a36d63813c03a1fb (127.24.249.130:44293)
I20260620 08:14:31.962217 31169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:55947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:31.976836 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40551
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:32.065259 31499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:32.065512 31499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:32.065542 31499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:32.066861 31499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:32.066918 31499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:32.066948 31499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:32.069123 31499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:32.069402 31499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:32.069600 31499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:32.075292 31505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:32.075273 31507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:32.075340 31504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:32.075668 31499 server_base.cc:1034] running on GCE node
I20260620 08:14:32.075968 31499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:32.076246 31499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:32.077464 31499 hybrid_clock.cc:648] HybridClock initialized: now 1781943272077435 us; error 44 us; skew 500 ppm
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.080389 31499 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:32.080924 31499 webserver.cc:460] Webserver started at http://127.24.249.131:39289/ using document root <none> and password file <none>
I20260620 08:14:32.081265 31499 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:32.081342 31499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:32.081478 31499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:32.082703 31499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "c39d55e1a8904b10b505dfea5e160859"
format_stamp: "Formatted at 2026-06-20 08:14:32 on dist-test-slave-65w8"
server_key: "c048904a23449a0e46bce8629568fb03"
server_key_iv: "bfb6827d44f766ff9d9f362805a8f0f8"
server_key_version: "encryptionkey@0"
I20260620 08:14:32.083081 31499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "c39d55e1a8904b10b505dfea5e160859"
format_stamp: "Formatted at 2026-06-20 08:14:32 on dist-test-slave-65w8"
server_key: "c048904a23449a0e46bce8629568fb03"
server_key_iv: "bfb6827d44f766ff9d9f362805a8f0f8"
server_key_version: "encryptionkey@0"
I20260620 08:14:32.084522 31499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:32.085333 31514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:32.085537 31499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:32.085631 31499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "c39d55e1a8904b10b505dfea5e160859"
format_stamp: "Formatted at 2026-06-20 08:14:32 on dist-test-slave-65w8"
server_key: "c048904a23449a0e46bce8629568fb03"
server_key_iv: "bfb6827d44f766ff9d9f362805a8f0f8"
server_key_version: "encryptionkey@0"
I20260620 08:14:32.085705 31499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:32.104038 31499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:32.106483 31499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:32.106635 31499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:32.106889 31499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:32.107220 31499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:32.107282 31499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:32.107318 31499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:32.107345 31499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:32.113286 31499 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:35243
I20260620 08:14:32.113345 31627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:35243 every 8 connection(s)
I20260620 08:14:32.113696 31499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:32.122597 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31499
I20260620 08:14:32.122699 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:32.122900 25574 external_mini_cluster.cc:1428] Setting key ea62ba60096eb0246c96c248bf42d129
I20260620 08:14:32.125182 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:32.114518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:48439 (local address 127.24.249.190:40551)
0620 08:14:32.114584 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:32.114586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:32.114814 (+ 228us) server_negotiation.cc:407] Connection header received
0620 08:14:32.115804 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:32.115805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:32.115818 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:32.115869 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:32.116690 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.117316 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.118550 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.118773 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.118862 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:32.121385 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:32.121387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:32.121387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:32.121395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:32.123504 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.124067 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.124068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.124069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.124178 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.124491 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.124492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.124492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.124621 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:32.124695 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:32.125027 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 08:14:32.125113 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:14:32.125376 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:32.114497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40551 (local address 127.24.249.131:48439)
0620 08:14:32.114678 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:32.114715 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:32.115670 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:32.115950 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:32.115954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:32.116342 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:32.116606 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:32.116611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.117412 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.117414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:32.118451 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:32.118454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.118875 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.118876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:32.118999 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:32.119641 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:32.119649 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:32.121234 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:32.123616 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:32.123626 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:32.123629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:32.123955 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:32.124257 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:32.124258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:32.124258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:32.124389 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:32.124719 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:32.124724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:32.124873 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 08:14:32.124976 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:14:32.125113 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260620 08:14:32.125927 31628 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40551
I20260620 08:14:32.126055 31628 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:32.126252 31628 heartbeater.cc:511] Master 127.24.249.190:40551 requested a full tablet report, sending...
I20260620 08:14:32.126736 31169 ts_manager.cc:194] Registered new tserver with Master: c39d55e1a8904b10b505dfea5e160859 (127.24.249.131:35243)
I20260620 08:14:32.127183 31169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:48439
I20260620 08:14:32.135442 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/test-admin.keytab.
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.151129 25574 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 08:14:32.153142 31635 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:32.152169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.24.249.129:36927)
0620 08:14:32.152387 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:32.152391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:32.152402 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:32.152428 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:32.152429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:32.152443 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:32.152514 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:32.153040 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57878: BlockingRecv error: recv got EOF from 127.0.0.1:57878 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":115,"threads_started":1}
W20260620 08:14:32.153191 25574 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5cc' of type FILE
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.154414 25574 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.155436 25574 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:32.163702 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:32.156028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.24.249.129:36927)
0620 08:14:32.156087 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:32.156089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:32.156147 (+ 58us) server_negotiation.cc:407] Connection header received
0620 08:14:32.156203 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:32.156204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:32.156214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:32.156263 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:32.156714 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.157517 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.158310 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.158649 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.158752 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:32.160988 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:32.160996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:32.160996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:32.161004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:32.162619 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.162988 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.162990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.162991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.163041 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.163245 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.163246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.163246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.163378 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:32.163496 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:32.163563 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:14:32.163627 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 08:14:32.165620 31635 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:32.165038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.24.249.129:36927)
0620 08:14:32.165097 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:32.165099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:32.165151 (+ 52us) server_negotiation.cc:407] Connection header received
0620 08:14:32.165249 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:32.165250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:32.165261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:32.165306 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:32.165573 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57900: BlockingRecv error: recv got EOF from 127.0.0.1:57900 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 08:14:32.165719 25574 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5cc' of type FILE
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.166716 25574 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.167630 25574 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:32.174322 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:32.168139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.24.249.129:36927)
0620 08:14:32.168207 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:32.168208 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:32.168216 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:32.168277 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:32.168277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:32.168285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:32.168340 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:32.168656 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.169133 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.169761 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.170009 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.170060 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:32.172040 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:32.172041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:32.172041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:32.172047 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:32.173184 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.173480 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.173480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.173480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.173527 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.173690 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.173690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.173691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.173777 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:32.173905 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:32.174100 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 08:14:32.174161 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 08:14:32.175864 25574 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781943243671421-25574-0/krb5kdc/krb5cc' of type FILE
W20260620 08:14:32.175894 31635 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:32.175276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.24.249.129:36927)
0620 08:14:32.175313 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:32.175315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:32.175421 (+ 106us) server_negotiation.cc:407] Connection header received
0620 08:14:32.175437 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:32.175438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:32.175446 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:32.175483 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:32.175845 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57906: BlockingRecv error: recv got EOF from 127.0.0.1:57906 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.176844 25574 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:32.177685 25574 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:14:32.185066 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:32.178295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.24.249.129:36927)
0620 08:14:32.178349 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:32.178351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:32.178436 (+ 85us) server_negotiation.cc:407] Connection header received
0620 08:14:32.178478 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:32.178479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:32.178487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:32.178535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:32.178947 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.179404 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.180137 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:32.180360 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:32.180429 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:32.182577 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:32.182579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:32.182579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:32.182586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:32.183862 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.184214 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.184215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.184216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.184272 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:32.184511 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:32.184512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:32.184512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:32.184647 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:32.184752 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:32.184884 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 08:14:32.184961 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 08:14:32.185712 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31231
I20260620 08:14:32.191839 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31365
I20260620 08:14:32.196622 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31499
I20260620 08:14:32.201282 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31137
2026-06-20T08:14:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3570 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31648](info): set up 2 sockets
Jun 20 08:14:32 dist-test-slave-65w8 krb5kdc[31648](info): commencing operation
krb5kdc: starting...
W20260620 08:14:33.250432 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab.
Entry for principal oryx/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab.
I20260620 08:14:33.377837 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40223
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab
--principal=oryx/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:40223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:33.461997 31661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:33.462270 31661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:33.462311 31661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:33.463624 31661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:33.463703 31661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:33.463732 31661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:33.463757 31661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:33.463788 31661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:33.465507 31661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:40223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab
--principal=oryx/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:33.465821 31661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:33.466094 31661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:33.471954 31666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:33.471954 31667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:33.471983 31669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:33.472699 31661 server_base.cc:1034] running on GCE node
I20260620 08:14:33.472972 31661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:33.473302 31661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:33.474502 31661 hybrid_clock.cc:648] HybridClock initialized: now 1781943273474487 us; error 60 us; skew 500 ppm
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:33.477700 31661 init.cc:377] Logged in from keytab as oryx/127.24.249.190@KRBTEST.COM (short username oryx)
I20260620 08:14:33.478206 31661 webserver.cc:460] Webserver started at http://127.24.249.190:42429/ using document root <none> and password file <none>
I20260620 08:14:33.478418 31661 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:33.478466 31661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:33.478556 31661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:33.479573 31661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "e1502f2ee3724eb980ad47e8a86d494b"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "3a74ec46497a547292215270662c879b"
server_key_iv: "3a6860b82589ddb7a39b2b86d1ea8586"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.480254 31661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "e1502f2ee3724eb980ad47e8a86d494b"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "3a74ec46497a547292215270662c879b"
server_key_iv: "3a6860b82589ddb7a39b2b86d1ea8586"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.481662 31661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:33.482911 31676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.483314 31661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:33.483395 31661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "e1502f2ee3724eb980ad47e8a86d494b"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "3a74ec46497a547292215270662c879b"
server_key_iv: "3a6860b82589ddb7a39b2b86d1ea8586"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.483459 31661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:33.495528 31661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:33.497990 31661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:33.498116 31661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:33.502526 31661 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:40223
I20260620 08:14:33.502561 31728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:40223 every 8 connection(s)
I20260620 08:14:33.502966 31661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:33.503330 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31661
I20260620 08:14:33.503455 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:33.503720 25574 external_mini_cluster.cc:1428] Setting key 105ec66c63507e58b80b785a4c06adb1
I20260620 08:14:33.504230 31729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:33.508131 31729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: Bootstrap starting.
I20260620 08:14:33.508791 31729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:33.509186 31729 log.cc:824] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:33.510442 31729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: No bootstrap required, opened a new log
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:33.512454 31729 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } }
I20260620 08:14:33.512602 31729 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:33.512625 31729 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1502f2ee3724eb980ad47e8a86d494b, State: Initialized, Role: FOLLOWER
I20260620 08:14:33.512782 31729 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } }
I20260620 08:14:33.512871 31729 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:33.512916 31729 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:33.512965 31729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:33.513561 31729 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } }
I20260620 08:14:33.513706 31729 leader_election.cc:302] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1502f2ee3724eb980ad47e8a86d494b; no voters:
I20260620 08:14:33.513939 31729 leader_election.cc:288] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:33.513973 31734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:33.514097 31734 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 1 LEADER]: Becoming Leader. State: Replica: e1502f2ee3724eb980ad47e8a86d494b, State: Running, Role: LEADER
I20260620 08:14:33.514180 31734 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } }
I20260620 08:14:33.514232 31729 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:33.514514 31734 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1502f2ee3724eb980ad47e8a86d494b. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } }
I20260620 08:14:33.514600 31734 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:33.514532 31736 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } }
I20260620 08:14:33.514866 31736 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:33.515636 31739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:33.516299 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:33.504348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.24.249.190:40223)
0620 08:14:33.504532 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:33.504543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:33.504584 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:14:33.505022 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:33.505028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:33.505087 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:33.505208 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:33.506301 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.507239 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.508414 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.508919 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.509023 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:33.512007 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:33.512012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:33.512012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:33.512024 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:33.514263 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:33.514703 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:33.514707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:33.514707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:33.514931 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:33.515187 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:33.515188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:33.515189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:33.515531 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:33.515703 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:33.515886 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 08:14:33.516048 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260620 08:14:33.516673 31739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:33.518718 31739 catalog_manager.cc:1347] Generated new cluster ID: 690e28b1718b448f95c84776431423b2
I20260620 08:14:33.518792 31739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:33.530257 31739 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:33.530800 31739 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:33.540460 31739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: Generated new TSK 0
I20260620 08:14:33.540684 31739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab.
Entry for principal oryx/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab.
I20260620 08:14:33.569943 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40223
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab
--principal=oryx/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:33.653453 31755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:33.653676 31755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:33.653702 31755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:33.654982 31755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:33.655026 31755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:33.655054 31755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:33.656536 31755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab
--principal=oryx/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:33.656767 31755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:33.657029 31755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:33.662016 31760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:33.662063 31763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:33.662120 31755 server_base.cc:1034] running on GCE node
W20260620 08:14:33.662012 31761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:33.662424 31755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:33.662616 31755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:33.663825 31755 hybrid_clock.cc:648] HybridClock initialized: now 1781943273663831 us; error 50 us; skew 500 ppm
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:33.666882 31755 init.cc:377] Logged in from keytab as oryx/127.24.249.129@KRBTEST.COM (short username oryx)
I20260620 08:14:33.667495 31755 webserver.cc:460] Webserver started at http://127.24.249.129:39583/ using document root <none> and password file <none>
I20260620 08:14:33.667753 31755 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:33.667804 31755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:33.667915 31755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:33.668938 31755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "044fd481d6e11f79cdde7efd5fe45603"
server_key_iv: "b11cf184089a3a3735183c32da74d8a8"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.669247 31755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "044fd481d6e11f79cdde7efd5fe45603"
server_key_iv: "b11cf184089a3a3735183c32da74d8a8"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.670665 31755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:33.671427 31770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.671686 31755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:33.671794 31755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "044fd481d6e11f79cdde7efd5fe45603"
server_key_iv: "b11cf184089a3a3735183c32da74d8a8"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.671875 31755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:33.686946 31755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:33.689823 31755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:33.690128 31755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:33.690392 31755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:33.690740 31755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:33.690780 31755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.690804 31755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:33.690816 31755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.697031 31755 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:34493
I20260620 08:14:33.697113 31883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:34493 every 8 connection(s)
I20260620 08:14:33.697458 31755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:33.705538 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31755
I20260620 08:14:33.705641 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:33.705977 25574 external_mini_cluster.cc:1428] Setting key 2e65feabfccb3553e7f454d775ce7c29
I20260620 08:14:33.707960 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:33.698347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:41327 (local address 127.24.249.190:40223)
0620 08:14:33.698452 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:33.698454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:33.698588 (+ 134us) server_negotiation.cc:407] Connection header received
0620 08:14:33.699278 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:33.699280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:33.699292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:33.699346 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:33.700354 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.700894 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.702059 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.702307 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.702373 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:33.704687 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:33.704688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:33.704688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:33.704697 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:33.706382 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:33.706904 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:33.706905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:33.706905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:33.706973 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:33.707295 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:33.707296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:33.707297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:33.707409 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:33.707466 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:33.707813 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 08:14:33.707897 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:14:33.708251 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:33.698290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.129:41327)
0620 08:14:33.698453 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:33.698498 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:33.699149 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:33.699410 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:33.699414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:33.700011 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:33.700224 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:33.700230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.701050 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.701051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:33.701969 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:33.701972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.702408 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.702410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:33.702521 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:33.703077 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:33.703086 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:33.704589 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:33.706476 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:33.706481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:33.706483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:33.706807 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:33.707062 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:33.707063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:33.707064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:33.707221 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:33.707479 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:33.707486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:33.707665 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 08:14:33.707784 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 08:14:33.707966 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260620 08:14:33.708796 31884 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40223
I20260620 08:14:33.708922 31884 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:33.709115 31884 heartbeater.cc:511] Master 127.24.249.190:40223 requested a full tablet report, sending...
I20260620 08:14:33.709658 31693 ts_manager.cc:194] Registered new tserver with Master: 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493)
I20260620 08:14:33.710318 31693 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:41327
WARNING: no policy specified for oryx/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab.
Entry for principal oryx/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab.
I20260620 08:14:33.733290 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40223
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab
--principal=oryx/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:33.815068 31889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:33.815274 31889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:33.815297 31889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:33.816499 31889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:33.816542 31889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:33.816568 31889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:33.818091 31889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab
--principal=oryx/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:33.818286 31889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:33.818465 31889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:33.823727 31894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:33.823714 31897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:33.823719 31895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:33.823894 31889 server_base.cc:1034] running on GCE node
I20260620 08:14:33.824218 31889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:33.824487 31889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:33.825687 31889 hybrid_clock.cc:648] HybridClock initialized: now 1781943273825687 us; error 44 us; skew 500 ppm
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:33.828488 31889 init.cc:377] Logged in from keytab as oryx/127.24.249.130@KRBTEST.COM (short username oryx)
I20260620 08:14:33.828949 31889 webserver.cc:460] Webserver started at http://127.24.249.130:39705/ using document root <none> and password file <none>
I20260620 08:14:33.829183 31889 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:33.829253 31889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:33.829408 31889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:33.830545 31889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "7df02488ec6943258258b470d8991b9b"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "a14bd73c8de545bc9b4cc5450c704197"
server_key_iv: "5699a8bebb92770c99aac25eb095dcaa"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.830924 31889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "7df02488ec6943258258b470d8991b9b"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "a14bd73c8de545bc9b4cc5450c704197"
server_key_iv: "5699a8bebb92770c99aac25eb095dcaa"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.832289 31889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:14:33.833171 31904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.833400 31889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:33.833518 31889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "7df02488ec6943258258b470d8991b9b"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "a14bd73c8de545bc9b4cc5450c704197"
server_key_iv: "5699a8bebb92770c99aac25eb095dcaa"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.833598 31889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:33.850049 31889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:33.852380 31889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:33.852576 31889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:33.852833 31889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:33.853184 31889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:33.853240 31889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.853298 31889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:33.853322 31889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.859162 31889 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:45567
I20260620 08:14:33.859213 32017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:45567 every 8 connection(s)
I20260620 08:14:33.859611 31889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:33.868654 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 31889
I20260620 08:14:33.868818 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:33.869015 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:33.860416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:53943 (local address 127.24.249.190:40223)
0620 08:14:33.860482 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:33.860484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:33.860687 (+ 203us) server_negotiation.cc:407] Connection header received
0620 08:14:33.861174 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:33.861175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:33.861186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:33.861240 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:33.862155 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.862663 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.863631 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.863817 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.863872 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:33.866187 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:33.866189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:33.866189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:33.866196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:33.867510 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:33.867942 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:33.867944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:33.867944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:33.867998 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:33.868242 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:33.868243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:33.868244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:33.868367 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:33.868431 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:33.868885 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 08:14:33.868950 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:33.869040 25574 external_mini_cluster.cc:1428] Setting key 8b61fd16a7cf6f96b166ef6f265a6bbd
I20260620 08:14:33.869215 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:33.860397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.130:53943)
0620 08:14:33.860570 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:33.860605 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:33.861092 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:33.861304 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:33.861307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:33.861886 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:33.862075 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:33.862079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.862754 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.862754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:33.863542 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:33.863544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:33.863918 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:33.863919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:33.864025 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:33.864503 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:33.864511 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:33.866103 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:33.867588 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:33.867594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:33.867596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:33.867870 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:33.868061 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:33.868061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:33.868061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:33.868165 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:33.868430 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:33.868434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:33.868623 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 08:14:33.868788 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 08:14:33.868974 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260620 08:14:33.869830 32018 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40223
I20260620 08:14:33.869968 32018 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:33.870112 32018 heartbeater.cc:511] Master 127.24.249.190:40223 requested a full tablet report, sending...
I20260620 08:14:33.870500 31689 ts_manager.cc:194] Registered new tserver with Master: 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567)
I20260620 08:14:33.870841 31689 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:53943
WARNING: no policy specified for oryx/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab.
Entry for principal oryx/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab.
I20260620 08:14:33.894654 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40223
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab
--principal=oryx/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:33.978323 32023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:33.978588 32023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:33.978628 32023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:33.979826 32023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:33.979902 32023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:33.979946 32023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:33.981441 32023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab
--principal=oryx/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:33.981725 32023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:33.981994 32023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:33.987329 32031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:33.987298 32028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:33.987311 32029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:33.987593 32023 server_base.cc:1034] running on GCE node
I20260620 08:14:33.987857 32023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:33.988131 32023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:33.989290 32023 hybrid_clock.cc:648] HybridClock initialized: now 1781943273989267 us; error 43 us; skew 500 ppm
Jun 20 08:14:33 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:33.991875 32023 init.cc:377] Logged in from keytab as oryx/127.24.249.131@KRBTEST.COM (short username oryx)
I20260620 08:14:33.992313 32023 webserver.cc:460] Webserver started at http://127.24.249.131:41795/ using document root <none> and password file <none>
I20260620 08:14:33.992508 32023 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:33.992548 32023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:33.992643 32023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:33.993616 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "23117b06df41450d8c1838e1bb225f1d"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "6b3c0b8c3e29db6fdf2d03694728958a"
server_key_iv: "1ec023e17da79a47b315bb914e1e5071"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.993984 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "23117b06df41450d8c1838e1bb225f1d"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "6b3c0b8c3e29db6fdf2d03694728958a"
server_key_iv: "1ec023e17da79a47b315bb914e1e5071"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.995265 32023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:33.996027 32038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:33.996222 32023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:33.996323 32023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "23117b06df41450d8c1838e1bb225f1d"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "6b3c0b8c3e29db6fdf2d03694728958a"
server_key_iv: "1ec023e17da79a47b315bb914e1e5071"
server_key_version: "encryptionkey@0"
I20260620 08:14:33.996378 32023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.008273 32023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.011329 32023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:34.011452 32023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.011646 32023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:34.011988 32023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:34.012025 32023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.012048 32023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:34.012094 32023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.018900 32023 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:41293
I20260620 08:14:34.018991 32151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:41293 every 8 connection(s)
I20260620 08:14:34.019313 32023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:34.020452 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 32023
I20260620 08:14:34.020547 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:34.020726 25574 external_mini_cluster.cc:1428] Setting key 411621a61403f145f50729436d02bfa0
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:34.029250 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.020139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:36401 (local address 127.24.249.190:40223)
0620 08:14:34.020229 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.020231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.020483 (+ 252us) server_negotiation.cc:407] Connection header received
0620 08:14:34.021140 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.021142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.021153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.021199 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.022302 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.022804 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.023850 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.024056 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.024139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.026406 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.026407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.026408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.026415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.027843 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.028274 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.028275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.028276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.028342 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.028596 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.028597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.028598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.028720 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.028796 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.029064 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.029146 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:14:34.029450 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.020080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:36401)
0620 08:14:34.020328 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.020374 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.021032 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.021278 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.021283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.021994 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.022190 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.022195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.022907 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.022908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.023713 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.023715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.024147 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.024148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.024280 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.024785 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.024793 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.026307 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.027952 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.027959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.027962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.028196 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.028411 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.028411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.028412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.028522 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.028786 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.028793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.028933 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.029043 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.029206 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260620 08:14:34.030011 32152 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40223
I20260620 08:14:34.030107 32152 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.030297 32152 heartbeater.cc:511] Master 127.24.249.190:40223 requested a full tablet report, sending...
I20260620 08:14:34.030763 31689 ts_manager.cc:194] Registered new tserver with Master: 23117b06df41450d8c1838e1bb225f1d (127.24.249.131:41293)
I20260620 08:14:34.031188 31689 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:36401
I20260620 08:14:34.032725 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.249.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.189.keytab.
Entry for principal oryx/127.24.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.189.keytab.
I20260620 08:14:34.062104 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:39301
--webserver_interface=127.24.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.189.keytab
--principal=oryx/127.24.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:40223,127.24.249.189:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:34.146701 32157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:34.146947 32157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:34.146984 32157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:34.148175 32157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:34.148247 32157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:34.148273 32157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:34.148304 32157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:34.148331 32157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:34.149849 32157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:40223,127.24.249.189:39301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.189.keytab
--principal=oryx/127.24.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
W20260620 08:14:34.150053 32157 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.249.190:40223,127.24.249.189:39301'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 08:14:34.150209 32157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:34.150444 32157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:34.155422 32163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.155431 32162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.155431 32165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.155701 32157 server_base.cc:1034] running on GCE node
I20260620 08:14:34.155999 32157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:34.156288 32157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:34.157467 32157 hybrid_clock.cc:648] HybridClock initialized: now 1781943274157449 us; error 28 us; skew 500 ppm
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:34.160049 32157 init.cc:377] Logged in from keytab as oryx/127.24.249.189@KRBTEST.COM (short username oryx)
I20260620 08:14:34.160490 32157 webserver.cc:460] Webserver started at http://127.24.249.189:39837/ using document root <none> and password file <none>
I20260620 08:14:34.160722 32157 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.160799 32157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.160948 32157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:34.162106 32157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/instance:
uuid: "684f14a55dc6408d9d7e9dd2048362ce"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "8a127a074a398d1a18b655e69098025a"
server_key_iv: "990015ac15f2db4009a5638a7a7838bc"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.162544 32157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal/instance:
uuid: "684f14a55dc6408d9d7e9dd2048362ce"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "8a127a074a398d1a18b655e69098025a"
server_key_iv: "990015ac15f2db4009a5638a7a7838bc"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.164006 32157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:34.164791 32172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.164965 32157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:34.165069 32157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
uuid: "684f14a55dc6408d9d7e9dd2048362ce"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "8a127a074a398d1a18b655e69098025a"
server_key_iv: "990015ac15f2db4009a5638a7a7838bc"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.165141 32157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.177399 32157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.179796 32157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:34.179960 32157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.184092 32157 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.189:39301
I20260620 08:14:34.184597 32157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
I20260620 08:14:34.184813 32224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.189:39301 every 8 connection(s)
I20260620 08:14:34.185588 32225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:34.187294 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 32157
I20260620 08:14:34.187150 32225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.189@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.249.189@KRBTEST.COM
I20260620 08:14:34.196664 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.188176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.24.249.190:40223)
0620 08:14:34.188220 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.188222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.188374 (+ 152us) server_negotiation.cc:407] Connection header received
0620 08:14:34.188589 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.188590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.188602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.188677 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.189225 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.189718 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.190690 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.190896 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.190982 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.193234 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.193236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.193236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.193245 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.195054 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.195489 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.195491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.195492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.195554 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.195899 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.195899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.195900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.196071 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.196145 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.196483 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.196569 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:34.196854 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.187901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53336)
0620 08:14:34.188258 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.188294 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.188499 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.188701 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.188703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.188978 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.189132 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.189136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.189829 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.189830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.190576 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.190578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.190991 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.190993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.191112 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.191636 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.191646 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.193135 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.195188 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.195194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.195196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.195406 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.195671 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.195672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.195672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.195822 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.196152 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.196157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.196338 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.196448 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.196569 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"spinlock_wait_cycles":384,"thread_start_us":104,"threads_started":1}
I20260620 08:14:34.197201 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.187901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.24.249.189:39301)
0620 08:14:34.188052 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.188061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.188098 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:14:34.188480 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.188486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.188505 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.188616 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.189223 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.190099 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.190824 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.191122 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.191197 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.193748 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.193755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.193756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.193763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.195725 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.196096 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.196099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.196100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.196159 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.196366 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.196367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.196367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.196537 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.196682 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.197043 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.197135 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260620 08:14:34.197623 32225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.189@KRBTEST.COM for oryx/127.24.249.189@KRBTEST.COM
I20260620 08:14:34.203794 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.197943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.0.0.1:39884)
0620 08:14:34.197977 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.197988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.198026 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.198141 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.198141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.198275 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.198305 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.198306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.198897 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.198898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.199438 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.199439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.199773 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.199774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.199838 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.200287 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.200292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.201463 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.202895 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.202896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.202896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.203059 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.203228 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.203229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.203229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.203280 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.203504 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.203505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.203541 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.203626 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.203704 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 08:14:34.203846 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.197967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.24.249.189:39301)
0620 08:14:34.198018 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.198020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.198028 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:34.198082 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.198083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.198090 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.198129 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.198361 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.198827 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.199501 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.199681 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.199734 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.201542 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.201543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.201544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.201550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.202817 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.203126 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.203127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.203127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.203172 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.203338 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.203339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.203339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.203429 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.203478 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.203651 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.203749 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":32}
I20260620 08:14:34.205241 32225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Bootstrap starting.
I20260620 08:14:34.206135 32225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:34.206377 32225 log.cc:824] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:34.207116 32225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: No bootstrap required, opened a new log
I20260620 08:14:34.208493 32225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } }
I20260620 08:14:34.208645 32225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:34.208669 32225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684f14a55dc6408d9d7e9dd2048362ce, State: Initialized, Role: FOLLOWER
I20260620 08:14:34.208861 32225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } }
I20260620 08:14:34.209249 32225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:34.209221 32231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } } }
I20260620 08:14:34.209338 32231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:14:34.211016 32242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:34.211078 32242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:34.212663 32157 master_runner.cc:428] Detected that this master 684f14a55dc6408d9d7e9dd2048362ce is joining an existing cluster
I20260620 08:14:34.212715 32157 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.249.189:39301
I20260620 08:14:34.220662 32248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.213742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53342)
0620 08:14:34.213937 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.213954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.214028 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.214155 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.214156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.214307 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.214383 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.214384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.215455 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.215456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.216139 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.216141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.216639 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.216640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.216696 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.217221 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.217225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.217541 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.219534 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.219535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.219535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.219758 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.219969 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.219970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.219971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.220036 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.220342 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.220343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.220385 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.220456 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.220551 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260620 08:14:34.220682 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.213823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.24.249.190:40223)
0620 08:14:34.213893 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.213895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.214002 (+ 107us) server_negotiation.cc:407] Connection header received
0620 08:14:34.214073 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.214074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.214084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.214149 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.214480 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.215368 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.216218 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.216521 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.216595 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.217634 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.217636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.217636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.217643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.219440 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.219832 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.219833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.219834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.219898 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.220116 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.220118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.220118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.220255 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.220317 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.220515 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.220609 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:34.224601 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.213918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.24.249.189:39301)
0620 08:14:34.213956 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.213958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.218053 (+ 4095us) server_negotiation.cc:407] Connection header received
0620 08:14:34.218118 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.218119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.218128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.218155 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.218495 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.219063 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.219772 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.220029 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.220120 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.221715 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.221717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.221717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.221723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.223476 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.223824 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.223825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.223826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.223876 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.224076 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.224077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.224078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.224185 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.224261 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.224383 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.224510 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":21}
I20260620 08:14:34.224613 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.213774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.0.0.1:39896)
0620 08:14:34.217973 (+ 4199us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.217984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.218043 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.218212 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.218213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.218373 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.218414 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.218415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.219154 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.219155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.219677 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.219678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.220224 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.220224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.220268 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.220834 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.220840 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.221621 (+ 781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.223576 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.223577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.223577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.223741 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.223949 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.223950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.223950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.224003 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.224251 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.224252 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.224289 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.224356 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.224461 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4178,"thread_start_us":58,"threads_started":1}
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.190@KRBTEST.COM for oryx/127.24.249.189@KRBTEST.COM
I20260620 08:14:34.231050 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.223219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.24.249.189:39301)
0620 08:14:34.223536 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.223538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.223545 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:34.223596 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.223597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.223606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.224178 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.224225 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.224977 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.225834 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.226168 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.226239 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.228440 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.228441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.228441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.228447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.229846 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.230329 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.230330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.230330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.230381 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.230573 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.230573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.230574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.230662 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.230734 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.230918 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.230976 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":86,"threads_started":1}
I20260620 08:14:34.231139 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.223152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.0.0.1:39904)
0620 08:14:34.223322 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.223363 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.223423 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.223694 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.223700 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.224041 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.224101 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.224103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.225084 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.225085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.225757 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.225760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.226262 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.226263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.226326 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.226822 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.226826 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.228354 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.229976 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.229983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.229985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.230250 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.230450 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.230451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.230451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.230517 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.230747 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.230754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.230807 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.230902 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.231019 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260620 08:14:34.231679 31689 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.249.189:39301: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce"
member_type: NON_VOTER
last_known_addr {
host: "127.24.249.189"
port: 39301
}
attrs {
promote: true
}
}
dest_uuid: "e1502f2ee3724eb980ad47e8a86d494b"
cas_config_opid_index: -1
I20260620 08:14:34.231993 31689 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } }
I20260620 08:14:34.232448 32253 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 684f14a55dc6408d9d7e9dd2048362ce (127.24.249.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } }
I20260620 08:14:34.232761 32199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:34.233469 32253 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.249.189:39301
I20260620 08:14:34.233635 32254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } }
I20260620 08:14:34.233683 32199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Refusing update from remote peer e1502f2ee3724eb980ad47e8a86d494b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 08:14:34.233707 32157 master.cc:537] Master@127.24.249.189:39301 shutting down...
I20260620 08:14:34.234771 32254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:34.234040 32252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Connected to new peer: Peer: permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:14:34.236299 31680 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 08:14:34.236522 32157 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:34.236624 32157 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:34.236693 32157 tablet_replica.cc:330] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: stopping tablet replica
W20260620 08:14:34.236752 31680 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b -> Peer 684f14a55dc6408d9d7e9dd2048362ce (127.24.249.189:39301): Couldn't send request to peer 684f14a55dc6408d9d7e9dd2048362ce. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 08:14:34.249576 32157 master.cc:552] Master@127.24.249.189:39301 shutdown complete.
I20260620 08:14:34.249634 32157 master_runner.cc:305] Clearing existing system tablet
I20260620 08:14:34.249804 32157 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.249852 32157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.250545 32157 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:34.251052 32258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.251180 32157 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.251237 32157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:34.251259 32157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
uuid: "684f14a55dc6408d9d7e9dd2048362ce"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "8a127a074a398d1a18b655e69098025a"
server_key_iv: "990015ac15f2db4009a5638a7a7838bc"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.251909 32157 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:34.253559 32157 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 08:14:34.253623 32157 log.cc:1196] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 08:14:34.254031 32157 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Deleting consensus metadata
I20260620 08:14:34.254168 32157 master_runner.cc:315] Copying system tablet from 127.24.249.190:40223
I20260620 08:14:34.254818 32157 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: tablet copy: Beginning tablet copy session from remote peer at address 127.24.249.190:40223
I20260620 08:14:34.261052 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.255207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.24.249.190:40223)
0620 08:14:34.255257 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.255259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.255323 (+ 64us) server_negotiation.cc:407] Connection header received
0620 08:14:34.255405 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.255406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.255416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.255474 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.255801 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.256298 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.257026 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.257241 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.257326 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.258360 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.258362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.258362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.258370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.259808 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.260200 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.260202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.260202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.260267 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.260465 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.260467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.260467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.260585 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.260715 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.260872 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.260967 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:14:34.261065 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.255142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53354)
0620 08:14:34.255256 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.255268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.255341 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.255481 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.255481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.255659 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.255706 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.255707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.256379 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.256380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.256939 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.256940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.257328 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.257329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.257397 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.257948 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.257953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.258268 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.259932 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.259933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.259934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.260120 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.260337 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.260338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.260338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.260403 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.260707 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.260708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.260756 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.260844 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.260953 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":70,"threads_started":1}
I20260620 08:14:34.261605 31713 tablet_copy_service.cc:140] P e1502f2ee3724eb980ad47e8a86d494b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 684f14a55dc6408d9d7e9dd2048362ce ({username='oryx', principal='oryx/127.24.249.189@KRBTEST.COM'} at 127.0.0.1:53354)
I20260620 08:14:34.261688 31713 tablet_copy_service.cc:161] P e1502f2ee3724eb980ad47e8a86d494b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 684f14a55dc6408d9d7e9dd2048362ce at {username='oryx', principal='oryx/127.24.249.189@KRBTEST.COM'} at 127.0.0.1:53354: session id = 684f14a55dc6408d9d7e9dd2048362ce-00000000000000000000000000000000
I20260620 08:14:34.262411 31713 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 08:14:34.263036 32157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 08:14:34.263955 32157 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: tablet copy: Starting download of 0 data blocks...
I20260620 08:14:34.264102 32157 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: tablet copy: Starting download of 1 WAL segments...
I20260620 08:14:34.264930 32157 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 08:14:34.265723 31713 tablet_copy_service.cc:342] P e1502f2ee3724eb980ad47e8a86d494b: Request end of tablet copy session 684f14a55dc6408d9d7e9dd2048362ce-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.249.189@KRBTEST.COM'} at 127.0.0.1:53354
I20260620 08:14:34.265784 31713 tablet_copy_service.cc:433] P e1502f2ee3724eb980ad47e8a86d494b: ending tablet copy session 684f14a55dc6408d9d7e9dd2048362ce-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 684f14a55dc6408d9d7e9dd2048362ce
I20260620 08:14:34.268247 32157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:34.269484 32268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.269527 32265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.269797 32266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.269837 32157 server_base.cc:1034] running on GCE node
I20260620 08:14:34.269975 32157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:34.270123 32157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:34.271288 32157 hybrid_clock.cc:648] HybridClock initialized: now 1781943274271263 us; error 37 us; skew 500 ppm
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:34.272787 32157 init.cc:377] Logged in from keytab as oryx/127.24.249.189@KRBTEST.COM (short username oryx)
I20260620 08:14:34.273106 32157 webserver.cc:460] Webserver started at http://127.24.249.189:46475/ using document root <none> and password file <none>
I20260620 08:14:34.273227 32157 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.273283 32157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.273936 32157 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:34.274475 32275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.274603 32157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:34.274667 32157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
uuid: "684f14a55dc6408d9d7e9dd2048362ce"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "8a127a074a398d1a18b655e69098025a"
server_key_iv: "990015ac15f2db4009a5638a7a7838bc"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.274721 32157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.281953 32157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.283488 32157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.286675 32157 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.189:39301
I20260620 08:14:34.287032 32327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.189:39301 every 8 connection(s)
I20260620 08:14:34.287129 32157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
I20260620 08:14:34.287815 32328 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:14:34.288251 32328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Bootstrap starting.
I20260620 08:14:34.290489 32328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:14:34.290723 32328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: Bootstrap complete.
I20260620 08:14:34.290942 32328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } }
I20260620 08:14:34.291011 32328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 684f14a55dc6408d9d7e9dd2048362ce, State: Initialized, Role: LEARNER
I20260620 08:14:34.291038 32328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } }
I20260620 08:14:34.291205 32328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:34.291219 32331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } }
I20260620 08:14:34.291320 32331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: LEARNER
I20260620 08:14:34.292627 32342 catalog_manager.cc:1259] Loaded cluster ID: 690e28b1718b448f95c84776431423b2
I20260620 08:14:34.292713 32342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: loading cluster ID for follower catalog manager: success
I20260620 08:14:34.293823 32342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: acquiring CA information for follower catalog manager: success
I20260620 08:14:34.294162 32342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:14:34.305420 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.298789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.24.249.189:39301)
0620 08:14:34.298901 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.298903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.298911 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:34.298940 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.298940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.298948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.299021 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.299585 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.300249 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.300959 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.301167 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.301252 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.302563 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.302564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.302565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.302574 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.304162 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.304584 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.304586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.304586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.304650 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.304855 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.304856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.304856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.304976 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.305082 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.305225 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.305311 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
WARNING: no policy specified for oryx/127.24.249.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.188.keytab.
Entry for principal oryx/127.24.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.188.keytab.
I20260620 08:14:34.333969 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:40053
--webserver_interface=127.24.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.188.keytab
--principal=oryx/127.24.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:34.419322 32346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:34.419523 32346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:34.419545 32346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:34.420720 32346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:34.420758 32346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:34.420773 32346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:34.420785 32346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:34.420833 32346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:34.422405 32346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.188.keytab
--principal=oryx/127.24.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:34.422646 32346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:34.422878 32346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:34.427819 32352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.427860 32351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.428078 32346 server_base.cc:1034] running on GCE node
W20260620 08:14:34.428081 32354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.428333 32346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:34.428655 32346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:34.429850 32346 hybrid_clock.cc:648] HybridClock initialized: now 1781943274429836 us; error 29 us; skew 500 ppm
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:34.432451 32346 init.cc:377] Logged in from keytab as oryx/127.24.249.188@KRBTEST.COM (short username oryx)
I20260620 08:14:34.432906 32346 webserver.cc:460] Webserver started at http://127.24.249.188:40405/ using document root <none> and password file <none>
I20260620 08:14:34.433161 32346 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.433233 32346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.433380 32346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:34.434499 32346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/instance:
uuid: "529cd900b5224f8cad12fd4ba462f268"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "1fecf49b79a0db886675cc04ce0e9187"
server_key_iv: "47c403360bae1b99028e2b03fa17318e"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.434896 32346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal/instance:
uuid: "529cd900b5224f8cad12fd4ba462f268"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "1fecf49b79a0db886675cc04ce0e9187"
server_key_iv: "47c403360bae1b99028e2b03fa17318e"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.436321 32346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:14:34.437341 32361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.437562 32346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:34.437692 32346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
uuid: "529cd900b5224f8cad12fd4ba462f268"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "1fecf49b79a0db886675cc04ce0e9187"
server_key_iv: "47c403360bae1b99028e2b03fa17318e"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.437779 32346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.457182 32346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.459462 32346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:34.459570 32346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.464285 32346 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.188:40053
I20260620 08:14:34.464411 32413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.188:40053 every 8 connection(s)
I20260620 08:14:34.464716 32346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
I20260620 08:14:34.465833 32414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:34.467402 32414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:14:34.469573 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 32346
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.188@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.249.188@KRBTEST.COM
I20260620 08:14:34.476912 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.468205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.24.249.190:40223)
0620 08:14:34.468274 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.468276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.468358 (+ 82us) server_negotiation.cc:407] Connection header received
0620 08:14:34.468885 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.468887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.468900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.468964 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.469626 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.470341 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.471308 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.471517 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.471601 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.473734 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.473736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.473736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.473744 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.475424 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.475835 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.475836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.475836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.475886 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.476178 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.476179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.476180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.476340 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.476410 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.476725 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.476806 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:14:34.476948 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.468031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53370)
0620 08:14:34.468204 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.468263 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.468776 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.469035 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.469039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.469284 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.469522 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.469526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.470432 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.470433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.471216 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.471218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.471626 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.471627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.471750 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.472267 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.472275 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.473645 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.475499 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.475505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.475508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.475741 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.475968 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.475969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.475969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.476102 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.476410 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.476416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.476570 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.476674 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.476790 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260620 08:14:34.477542 32414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:14:34.478691 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.470114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.24.249.188:40053)
0620 08:14:34.470270 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.470275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.470301 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:14:34.470320 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.470322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.470337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.470425 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.471004 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.471701 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.472451 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.472743 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.472850 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.475504 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.475511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.475511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.475518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.477434 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.477789 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.477798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.477798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.477883 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.478102 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.478103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.478103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.478260 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.478431 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.478516 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.478589 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.188@KRBTEST.COM for oryx/127.24.249.189@KRBTEST.COM
I20260620 08:14:34.483973 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.477799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.0.0.1:39930)
0620 08:14:34.477835 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.477846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.477908 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.478038 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.478038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.478178 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.478231 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.478233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.479003 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.479004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.479690 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.479692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.480029 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.480030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.480078 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.480500 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.480504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.481625 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.483116 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.483117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.483117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.483273 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.483497 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.483498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.483498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.483544 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.483754 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.483755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.483788 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.483833 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.483900 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 08:14:34.483980 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.477849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.24.249.189:39301)
0620 08:14:34.477904 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.477906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.477915 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:34.477972 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.477973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.477984 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.478023 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.478325 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.478885 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.479762 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.479967 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.480022 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.481703 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.481705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.481705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.481711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.483027 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.483365 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.483366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.483366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.483422 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.483610 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.483610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.483610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.483717 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.483764 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.483851 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.483919 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:34.484495 32414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.188@KRBTEST.COM for oryx/127.24.249.188@KRBTEST.COM
I20260620 08:14:34.490825 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.484800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.0.0.1:57750)
0620 08:14:34.484843 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.484853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.484888 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.484985 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.484986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.485119 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.485148 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.485149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.485724 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.485725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.486254 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.486256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.486757 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.486758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.486804 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.487235 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.487241 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.488487 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.489991 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.489992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.489993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.490160 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.490348 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.490348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.490349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.490396 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.490598 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.490599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.490630 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.490679 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.490746 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 08:14:34.490882 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.484834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.24.249.188:40053)
0620 08:14:34.484865 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.484867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.484877 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:34.484927 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.484928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.484935 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.484981 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.485190 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.485639 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.486331 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.486609 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.486707 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.488569 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.488570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.488570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.488576 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.489827 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.490224 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.490225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.490225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.490274 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.490451 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.490451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.490451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.490548 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.490598 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.490694 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.490788 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":15}
I20260620 08:14:34.492339 32414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Bootstrap starting.
I20260620 08:14:34.493306 32414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:34.493579 32414 log.cc:824] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:34.494377 32414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: No bootstrap required, opened a new log
I20260620 08:14:34.496116 32414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } }
I20260620 08:14:34.496263 32414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:34.496326 32414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529cd900b5224f8cad12fd4ba462f268, State: Initialized, Role: FOLLOWER
I20260620 08:14:34.496464 32414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } }
I20260620 08:14:34.496768 32414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:34.496732 32419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } } }
I20260620 08:14:34.496882 32419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:14:34.498384 32430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:34.498443 32430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:34.500638 32346 master_runner.cc:428] Detected that this master 529cd900b5224f8cad12fd4ba462f268 is joining an existing cluster
I20260620 08:14:34.500684 32346 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.249.188:40053
I20260620 08:14:34.508590 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.501763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.24.249.190:40223)
0620 08:14:34.501897 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.501900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.502065 (+ 165us) server_negotiation.cc:407] Connection header received
0620 08:14:34.502163 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.502164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.502175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.502206 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.502675 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.503397 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504432 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.504709 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504749 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.505989 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.505991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.505991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.505998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.507433 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.507898 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.507899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.507899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.507952 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.508159 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.508160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.508160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.508255 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.508308 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.508476 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.508533 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 08:14:34.508667 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.501694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53386)
0620 08:14:34.501978 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.501992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.502066 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.502280 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.502281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.502518 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.502586 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.502588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.503493 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.503493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.504339 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.504341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.504835 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.504892 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.505445 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.505450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.505910 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.507516 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.507517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.507517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.507734 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.508020 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.508021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.508022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.508096 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.508321 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.508322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.508378 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.508459 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.508560 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943273, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.190@KRBTEST.COM for oryx/127.24.249.188@KRBTEST.COM
I20260620 08:14:34.526223 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.510668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.0.0.1:57774)
0620 08:14:34.510727 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.510739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.510784 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.511058 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.511058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.511179 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.511214 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.511215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.512157 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.512157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.512645 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.512646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.513717 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.513718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.513768 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.514260 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.514265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.515748 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.525006 (+ 9258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.525007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.525008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.525279 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.525504 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.525505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.525506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.525563 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.525843 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.525844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.525921 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.525970 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.526093 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 08:14:34.526698 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.510742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.24.249.188:40053)
0620 08:14:34.510927 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.510929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.510937 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:34.510970 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.510971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.510981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.511055 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.511284 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.512076 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.512725 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.513633 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.513710 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.515834 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.515836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.515837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.515844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.524877 (+ 9033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.525368 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.525368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.525369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.525433 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.525632 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.525633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.525633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.525768 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.526415 (+ 647us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.526456 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.526597 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260620 08:14:34.527606 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.501977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.0.0.1:57766)
0620 08:14:34.502115 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.502126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.502203 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.502360 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.502360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.502635 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.502700 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.502702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.503589 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.503590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.504470 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.504472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.504944 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.505007 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.522956 (+ 17949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.522964 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.523460 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.526379 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.526380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.526381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.526637 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.526845 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.526846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.526846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.526895 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.527193 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.527194 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.527257 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.527334 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.527445 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":43,"thread_start_us":71,"threads_started":1}
I20260620 08:14:34.527616 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.501928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.24.249.188:40053)
0620 08:14:34.501962 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.501964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.502178 (+ 214us) server_negotiation.cc:407] Connection header received
0620 08:14:34.502268 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.502269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.502277 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.502306 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.502782 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.503493 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504558 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.504834 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504873 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.523602 (+ 18729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.523604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.523604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.523613 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.526034 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.526725 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.526726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.526727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.526776 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.526967 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.526968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.526969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.527096 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.527177 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.527369 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.527504 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":17}
I20260620 08:14:34.527927 31689 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.249.188:40053: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "529cd900b5224f8cad12fd4ba462f268"
member_type: NON_VOTER
last_known_addr {
host: "127.24.249.188"
port: 40053
}
attrs {
promote: true
}
}
dest_uuid: "e1502f2ee3724eb980ad47e8a86d494b"
cas_config_opid_index: 5
I20260620 08:14:34.528146 31689 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } }
I20260620 08:14:34.528993 32254 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.530097 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31755
I20260620 08:14:34.530331 32254 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.249.188:40053
I20260620 08:14:34.530397 32254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.530447 32254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:34.534304 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.528559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.0.0.1:39954)
0620 08:14:34.528645 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.528657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.528690 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.529178 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.529180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.529182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:34.529234 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.529235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.531750 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.531751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.533015 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.533018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.534002 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.534003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.534080 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.534111 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.534141 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.534188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 08:14:34.534680 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.528861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.24.249.189:39301)
0620 08:14:34.528994 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.528996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.529004 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:34.529028 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.529029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.529053 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.529135 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:34.529417 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.531657 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.533124 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.533850 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.534424 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.534500 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.534521 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.534572 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260620 08:14:34.534945 32302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 LEARNER]: Refusing update from remote peer e1502f2ee3724eb980ad47e8a86d494b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 08:14:34.534948 32346 master.cc:537] Master@127.24.249.188:40053 shutting down...
I20260620 08:14:34.535324 32252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Connected to new peer: Peer: permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260620 08:14:34.535967 31677 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b -> Peer 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188:40053): Couldn't send request to peer 529cd900b5224f8cad12fd4ba462f268. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 08:14:34.535928 32302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.537155 32331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1502f2ee3724eb980ad47e8a86d494b. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.537266 32331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: LEARNER
I20260620 08:14:34.537307 32331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: NON_VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: true } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.537364 32331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: LEARNER
I20260620 08:14:34.537928 32252 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: attempting to promote NON_VOTER 684f14a55dc6408d9d7e9dd2048362ce to VOTER
I20260620 08:14:34.538158 32252 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } }
I20260620 08:14:34.538324 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:34493
--local_ip_for_outbound_sockets=127.24.249.129
--tserver_master_addrs=127.24.249.190:40223
--webserver_port=39583
--webserver_interface=127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab
--principal=oryx/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:14:34.538575 32346 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:34.538661 32346 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:34.538704 32346 tablet_replica.cc:330] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: stopping tablet replica
I20260620 08:14:34.538687 32302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 LEARNER]: Refusing update from remote peer e1502f2ee3724eb980ad47e8a86d494b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 08:14:34.538865 32253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Connected to new peer: Peer: permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 08:14:34.539535 31677 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b -> Peer 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188:40053): Couldn't send request to peer 529cd900b5224f8cad12fd4ba462f268. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 08:14:34.539930 32254 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 684f14a55dc6408d9d7e9dd2048362ce (127.24.249.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.540877 32254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.540956 32254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:34.541018 32254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.541074 32254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:34.541329 32302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 684f14a55dc6408d9d7e9dd2048362ce (127.24.249.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.542279 32331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.542356 32331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:14:34.542520 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.501707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.24.249.189:39301)
0620 08:14:34.501758 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.501760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.502233 (+ 473us) server_negotiation.cc:407] Connection header received
0620 08:14:34.502339 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.502340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.502352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.502383 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.502839 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.503572 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504560 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.504850 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504887 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.525245 (+ 20358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.525248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.525248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.525258 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.527216 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.540882 (+ 13666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.540884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.540884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.540953 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.541737 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.541738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.541739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.541921 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.541983 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.542189 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.542294 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:14:34.542972 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.502080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.0.0.1:39944)
0620 08:14:34.502178 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.502187 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.502255 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.502519 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.502520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.502708 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.502760 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.502761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.503658 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.503658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.504470 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.504472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.504952 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.504952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.505007 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.524436 (+ 19429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.524441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.525117 (+ 676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.540449 (+ 15332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.540450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.540451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.540749 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.541575 (+ 826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.541576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.541577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.541658 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.542020 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.542021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.542074 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.542443 (+ 369us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.542825 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260620 08:14:34.552507 32346 master.cc:552] Master@127.24.249.188:40053 shutdown complete.
I20260620 08:14:34.552565 32346 master_runner.cc:305] Clearing existing system tablet
I20260620 08:14:34.552696 32346 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.552740 32346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.553489 32346 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:34.554198 32445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.554366 32346 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.554428 32346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:34.554451 32346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
uuid: "529cd900b5224f8cad12fd4ba462f268"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "1fecf49b79a0db886675cc04ce0e9187"
server_key_iv: "47c403360bae1b99028e2b03fa17318e"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.555239 32346 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:34.556710 32346 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 08:14:34.556779 32346 log.cc:1196] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 08:14:34.556982 32346 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Deleting consensus metadata
I20260620 08:14:34.557111 32346 master_runner.cc:315] Copying system tablet from 127.24.249.190:40223
I20260620 08:14:34.557744 32346 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: tablet copy: Beginning tablet copy session from remote peer at address 127.24.249.190:40223
I20260620 08:14:34.565176 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.558142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.24.249.190:40223)
0620 08:14:34.558192 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.558194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.558255 (+ 61us) server_negotiation.cc:407] Connection header received
0620 08:14:34.558347 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.558347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.558358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.558432 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.558820 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.559607 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.560714 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.561029 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.561068 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.562361 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.562363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.562364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.562371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.563981 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.564377 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.564378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.564378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.564437 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.564624 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.564624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.564625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.564747 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.564787 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.565004 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.565080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:14:34.565510 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.558066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53400)
0620 08:14:34.558199 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.558210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.558268 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.558448 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.558449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.558652 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.558736 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.558738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.559705 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.559706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.560611 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.560614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.561145 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.561146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.561209 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.561758 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.561763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.562257 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.564074 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.564075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.564076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.564293 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.564505 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.564506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.564506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.564559 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.564842 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.564843 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.564898 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.565305 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.565412 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260620 08:14:34.565783 31713 tablet_copy_service.cc:140] P e1502f2ee3724eb980ad47e8a86d494b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 529cd900b5224f8cad12fd4ba462f268 ({username='oryx', principal='oryx/127.24.249.188@KRBTEST.COM'} at 127.0.0.1:53400)
I20260620 08:14:34.565855 31713 tablet_copy_service.cc:161] P e1502f2ee3724eb980ad47e8a86d494b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 529cd900b5224f8cad12fd4ba462f268 at {username='oryx', principal='oryx/127.24.249.188@KRBTEST.COM'} at 127.0.0.1:53400: session id = 529cd900b5224f8cad12fd4ba462f268-00000000000000000000000000000000
I20260620 08:14:34.566330 31713 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 08:14:34.566830 32346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 08:14:34.568153 32346 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: tablet copy: Starting download of 0 data blocks...
I20260620 08:14:34.568285 32346 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: tablet copy: Starting download of 1 WAL segments...
I20260620 08:14:34.569108 32346 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 08:14:34.570122 31713 tablet_copy_service.cc:342] P e1502f2ee3724eb980ad47e8a86d494b: Request end of tablet copy session 529cd900b5224f8cad12fd4ba462f268-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.249.188@KRBTEST.COM'} at 127.0.0.1:53400
I20260620 08:14:34.570173 31713 tablet_copy_service.cc:433] P e1502f2ee3724eb980ad47e8a86d494b: ending tablet copy session 529cd900b5224f8cad12fd4ba462f268-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 529cd900b5224f8cad12fd4ba462f268
I20260620 08:14:34.573027 32346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:34.574380 32452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.574472 32455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.574769 32346 server_base.cc:1034] running on GCE node
W20260620 08:14:34.574820 32453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.574937 32346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:34.575080 32346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:34.576196 32346 hybrid_clock.cc:648] HybridClock initialized: now 1781943274576174 us; error 39 us; skew 500 ppm
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:34.577780 32346 init.cc:377] Logged in from keytab as oryx/127.24.249.188@KRBTEST.COM (short username oryx)
I20260620 08:14:34.578090 32346 webserver.cc:460] Webserver started at http://127.24.249.188:40849/ using document root <none> and password file <none>
I20260620 08:14:34.578224 32346 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.578260 32346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.579038 32346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:34.579651 32462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.579823 32346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:34.579880 32346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
uuid: "529cd900b5224f8cad12fd4ba462f268"
format_stamp: "Formatted at 2026-06-20 08:14:34 on dist-test-slave-65w8"
server_key: "1fecf49b79a0db886675cc04ce0e9187"
server_key_iv: "47c403360bae1b99028e2b03fa17318e"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.579945 32346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.610522 32346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.612514 32346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.616025 32346 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.188:40053
I20260620 08:14:34.616482 32346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
I20260620 08:14:34.616755 32514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.188:40053 every 8 connection(s)
I20260620 08:14:34.617645 32515 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:14:34.618285 32515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Bootstrap starting.
I20260620 08:14:34.621719 32515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:14:34.622038 32515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Bootstrap complete.
I20260620 08:14:34.622284 32515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } }
I20260620 08:14:34.622388 32515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 529cd900b5224f8cad12fd4ba462f268, State: Initialized, Role: LEARNER
I20260620 08:14:34.622432 32515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } }
I20260620 08:14:34.622648 32515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:34.622637 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:34.622985 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:14:34.624325 32529 catalog_manager.cc:1259] Loaded cluster ID: 690e28b1718b448f95c84776431423b2
I20260620 08:14:34.624408 32529 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: loading cluster ID for follower catalog manager: success
I20260620 08:14:34.625723 32529 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: acquiring CA information for follower catalog manager: success
I20260620 08:14:34.626048 32529 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 08:14:34.638442 32443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:34.638648 32443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:34.638671 32443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:34.639853 32443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:34.639894 32443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:34.639922 32443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:34.641418 32443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab
--principal=oryx/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:34493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=39583
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:34.641677 32443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:34.641996 32443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:14:34.647116 32443 server_base.cc:1034] running on GCE node
W20260620 08:14:34.647112 32535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.647249 32534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.647303 32537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.647544 32443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:34.647836 32443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:34.649027 32443 hybrid_clock.cc:648] HybridClock initialized: now 1781943274649014 us; error 27 us; skew 500 ppm
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:34.651859 32443 init.cc:377] Logged in from keytab as oryx/127.24.249.129@KRBTEST.COM (short username oryx)
I20260620 08:14:34.652516 32443 webserver.cc:460] Webserver started at http://127.24.249.129:39583/ using document root <none> and password file <none>
I20260620 08:14:34.652812 32443 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.652905 32443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.654807 32443 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:34.655566 32544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.655809 32443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:34.655918 32443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "044fd481d6e11f79cdde7efd5fe45603"
server_key_iv: "b11cf184089a3a3735183c32da74d8a8"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.656273 32443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.666121 32443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.668424 32443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:34.668587 32443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.668824 32443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:34.669131 32443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:34.669185 32443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.669222 32443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:34.669245 32443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.675570 32443 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:34493
I20260620 08:14:34.675609 32657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:34493 every 8 connection(s)
I20260620 08:14:34.676003 32443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:34.676190 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 32443
I20260620 08:14:34.676352 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31889
I20260620 08:14:34.681474 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:45567
--local_ip_for_outbound_sockets=127.24.249.130
--tserver_master_addrs=127.24.249.190:40223
--webserver_port=39705
--webserver_interface=127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab
--principal=oryx/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.188@KRBTEST.COM
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.189@KRBTEST.COM
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:34.692629 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.680717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:33917 (local address 127.24.249.189:39301)
0620 08:14:34.680770 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.680772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.680781 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:34.681143 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.681144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.681157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.681189 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.682338 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.683125 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.684756 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.685061 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.685102 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.688975 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.688976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.688977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.688985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.690870 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.691388 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.691389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.691390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.691461 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.691770 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.691771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.691771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.691965 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.692043 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.692380 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.692474 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:14:34.692860 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.680054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.24.249.129:33917)
0620 08:14:34.680512 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.680521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.681043 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.681718 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.681719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.681946 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.682224 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.682229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.683265 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.683266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.684645 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.684648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.685204 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.685205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.685350 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.686173 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.686177 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.688854 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.690965 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.690972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.690975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.691295 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.691548 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.691549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.691549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.691695 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.692031 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.692037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.692212 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.692341 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.692526 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":79,"thread_start_us":88,"threads_started":1}
I20260620 08:14:34.693428 32660 heartbeater.cc:347] Connected to a master server at 127.24.249.189:39301
I20260620 08:14:34.693537 32660 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.693763 32660 heartbeater.cc:511] Master 127.24.249.189:39301 requested a full tablet report, sending...
I20260620 08:14:34.694150 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.680775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:51295 (local address 127.24.249.190:40223)
0620 08:14:34.680913 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.680915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.680924 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:34.681254 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.681255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.681267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.681300 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.682338 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.683125 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.684756 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.685061 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.685102 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.690002 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.690003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.690004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.690011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.691698 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.693247 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.693249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.693249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.693314 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.693520 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.693521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.693521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.693661 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.693704 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.693967 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.694056 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 08:14:34.694442 32292 ts_manager.cc:194] Registered new tserver with Master: 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493)
I20260620 08:14:34.694681 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.680054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.129:51295)
0620 08:14:34.680582 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.680591 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.681168 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.681375 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.681376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.682165 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.682223 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.682229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.683265 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.683266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.684645 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.684648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.685203 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.685204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.685350 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.686080 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.686109 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.689927 (+ 3818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.692516 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.692517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.692517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.693150 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.693388 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.693389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.693389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.693445 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.693752 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.693753 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.693809 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.694287 (+ 478us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.694396 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":327,"thread_start_us":60,"threads_started":1}
I20260620 08:14:34.695056 32659 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40223
I20260620 08:14:34.695116 32659 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.695209 32659 heartbeater.cc:511] Master 127.24.249.190:40223 requested a full tablet report, sending...
I20260620 08:14:34.695515 31689 ts_manager.cc:194] Re-registered known tserver with Master: 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493)
I20260620 08:14:34.695922 31689 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:51295
I20260620 08:14:34.696107 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.680054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.24.249.129:44141)
0620 08:14:34.680232 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.680278 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.681021 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.681361 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.681365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.681927 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.682223 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.682229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.683484 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.683486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.684645 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.684648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.685212 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.685213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.685350 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.686235 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.686241 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.689512 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.693256 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.693257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.693257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.693471 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.693785 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.693786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.693786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.693987 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.695757 (+ 1770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.695758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.695809 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.695895 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.696000 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":12,"thread_start_us":99,"threads_started":1}
I20260620 08:14:34.696389 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.680343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:44141 (local address 127.24.249.188:40053)
0620 08:14:34.680455 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.680457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.680468 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:34.681166 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.681168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.681181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.681243 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.682332 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.683383 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.684742 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.685082 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.685151 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.689614 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.689616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.689616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.689625 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.692433 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.693661 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.693662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.693663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.693718 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.694061 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.694062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.694062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.694436 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.695766 (+ 1330us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.695935 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.696077 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":45,"threads_started":1}
I20260620 08:14:34.696699 32658 heartbeater.cc:347] Connected to a master server at 127.24.249.188:40053
I20260620 08:14:34.696748 32658 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.696815 32658 heartbeater.cc:511] Master 127.24.249.188:40053 requested a full tablet report, sending...
I20260620 08:14:34.697122 32479 ts_manager.cc:194] Registered new tserver with Master: 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493)
W20260620 08:14:34.769610 32666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:34.769829 32666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:34.769853 32666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:34.771085 32666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:34.771138 32666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:34.771166 32666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:34.772661 32666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab
--principal=oryx/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:45567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=39705
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:34.772913 32666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:34.773180 32666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:34.778618 32674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.778641 32671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.778640 32672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.778841 32666 server_base.cc:1034] running on GCE node
I20260620 08:14:34.779110 32666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:34.779412 32666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:34.780586 32666 hybrid_clock.cc:648] HybridClock initialized: now 1781943274780572 us; error 28 us; skew 500 ppm
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:34.783402 32666 init.cc:377] Logged in from keytab as oryx/127.24.249.130@KRBTEST.COM (short username oryx)
I20260620 08:14:34.783887 32666 webserver.cc:460] Webserver started at http://127.24.249.130:39705/ using document root <none> and password file <none>
I20260620 08:14:34.784159 32666 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.784243 32666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.785679 32666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:14:34.786428 32681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.786659 32666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:34.786779 32666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "7df02488ec6943258258b470d8991b9b"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "a14bd73c8de545bc9b4cc5450c704197"
server_key_iv: "5699a8bebb92770c99aac25eb095dcaa"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.787104 32666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.794833 32666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.796787 32666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:34.796897 32666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.797070 32666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:34.797395 32666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:34.797453 32666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.797493 32666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:34.797521 32666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.803401 32666 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:45567
I20260620 08:14:34.803469 326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:45567 every 8 connection(s)
I20260620 08:14:34.803820 32666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:34.807442 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 32666
I20260620 08:14:34.807606 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 32023
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.189@KRBTEST.COM
I20260620 08:14:34.813375 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:41293
--local_ip_for_outbound_sockets=127.24.249.131
--tserver_master_addrs=127.24.249.190:40223
--webserver_port=41795
--webserver_interface=127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab
--principal=oryx/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:42223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.188@KRBTEST.COM
I20260620 08:14:34.818019 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.804570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:48811 (local address 127.24.249.190:40223)
0620 08:14:34.804628 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.804630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.804906 (+ 276us) server_negotiation.cc:407] Connection header received
0620 08:14:34.806176 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.806177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.806192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.806252 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.807178 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.808015 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.809776 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.810110 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.810175 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.813702 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.813704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.813704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.813713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.816380 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.816796 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.816797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.816798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.816858 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.817052 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.817053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.817053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.817293 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.817360 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.817812 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.817918 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:14:34.818019 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.804612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:52587 (local address 127.24.249.189:39301)
0620 08:14:34.804653 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.804655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.805089 (+ 434us) server_negotiation.cc:407] Connection header received
0620 08:14:34.806174 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.806176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.806188 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.806217 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.807177 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.807978 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.809740 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.810101 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.810171 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.814099 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.814100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.814101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.814109 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.816163 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.816693 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.816694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.816695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.816752 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.817042 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.817043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.817043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.817330 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.817376 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.817803 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.817918 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:34.818608 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.804565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.24.249.130:52587)
0620 08:14:34.805028 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.805037 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.806063 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.806340 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.806344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.806788 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.807065 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.807071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.808237 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.808238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.809643 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.809645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.810235 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.810236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.810383 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.811007 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.811017 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.813999 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.816263 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.816270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.816272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.816600 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.816828 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.816829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.816829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.816964 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.817422 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.817429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.817610 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.818122 (+ 512us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.818398 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":166,"thread_start_us":85,"threads_started":1}
I20260620 08:14:34.818627 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.804565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.130:48811)
0620 08:14:34.804753 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.804788 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.806063 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.806340 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.806344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.806788 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.807065 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.807071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.808675 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.808676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.809643 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.809645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.810238 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.810239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.810383 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.811071 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.811077 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.813615 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.816465 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.816465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.816466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.816709 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.816929 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.816929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.816930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.816979 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.817421 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.817429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.817610 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.818147 (+ 537us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.818352 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":14,"thread_start_us":79,"threads_started":1}
I20260620 08:14:34.818660 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.804958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:43471 (local address 127.24.249.188:40053)
0620 08:14:34.805001 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.805003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.806041 (+ 1038us) server_negotiation.cc:407] Connection header received
0620 08:14:34.806198 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.806199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.806213 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.806252 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.807188 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.808020 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.811883 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.812208 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.812257 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.814486 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.814487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.814488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.814496 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.817190 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.817622 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.817623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.817624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.817683 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.817946 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.817947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.817947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.818093 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.818158 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.818413 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.818513 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:34.819089 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.804896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.24.249.130:43471)
0620 08:14:34.805953 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.805963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.806076 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.806357 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.806358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.806961 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.807064 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.807069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.810465 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.810466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.811761 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.811764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.812336 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.812337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.812402 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.812855 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.812860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.814392 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.817284 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.817285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.817286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.817529 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.817761 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.817762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.817762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.817828 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.818249 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.818250 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.818305 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.818785 (+ 480us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.818988 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":72,"threads_started":1}
I20260620 08:14:34.819319 329 heartbeater.cc:347] Connected to a master server at 127.24.249.189:39301
I20260620 08:14:34.819411 329 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.819474 327 heartbeater.cc:347] Connected to a master server at 127.24.249.188:40053
I20260620 08:14:34.819516 327 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.819591 329 heartbeater.cc:511] Master 127.24.249.189:39301 requested a full tablet report, sending...
I20260620 08:14:34.819597 327 heartbeater.cc:511] Master 127.24.249.188:40053 requested a full tablet report, sending...
I20260620 08:14:34.820027 32479 ts_manager.cc:194] Registered new tserver with Master: 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567)
I20260620 08:14:34.820052 32292 ts_manager.cc:194] Registered new tserver with Master: 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567)
I20260620 08:14:34.820451 328 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40223
I20260620 08:14:34.820504 328 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.820561 328 heartbeater.cc:511] Master 127.24.249.190:40223 requested a full tablet report, sending...
I20260620 08:14:34.820784 31689 ts_manager.cc:194] Re-registered known tserver with Master: 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567)
I20260620 08:14:34.821187 31689 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:48811
W20260620 08:14:34.900077 334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:34.900327 334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:34.900367 334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:34.901551 334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:34.901618 334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:34.901664 334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:34.903280 334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:42223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab
--principal=oryx/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:41293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=41795
--tserver_master_addrs=127.24.249.190:40223,127.24.249.189:39301,127.24.249.188:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:34.903570 334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:34.903855 334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:34.909024 339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:34.909073 340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.909233 334 server_base.cc:1034] running on GCE node
W20260620 08:14:34.909365 342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:34.909614 334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:34.909911 334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:34.911090 334 hybrid_clock.cc:648] HybridClock initialized: now 1781943274911069 us; error 47 us; skew 500 ppm
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:34.913762 334 init.cc:377] Logged in from keytab as oryx/127.24.249.131@KRBTEST.COM (short username oryx)
I20260620 08:14:34.914288 334 webserver.cc:460] Webserver started at http://127.24.249.131:41795/ using document root <none> and password file <none>
I20260620 08:14:34.914551 334 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:34.914604 334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:34.916057 334 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:34.916759 349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.916988 334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:34.917084 334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "23117b06df41450d8c1838e1bb225f1d"
format_stamp: "Formatted at 2026-06-20 08:14:33 on dist-test-slave-65w8"
server_key: "6b3c0b8c3e29db6fdf2d03694728958a"
server_key_iv: "1ec023e17da79a47b315bb914e1e5071"
server_key_version: "encryptionkey@0"
I20260620 08:14:34.917449 334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:34.934280 334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:34.936546 334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:34.936703 334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:34.937014 334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:34.937345 334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:34.937378 334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.937400 334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:34.937443 334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:34.943275 334 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:41293
I20260620 08:14:34.943362 462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:41293 every 8 connection(s)
I20260620 08:14:34.943710 334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:34.949980 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 334
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.188@KRBTEST.COM
Jun 20 08:14:34 dist-test-slave-65w8 krb5kdc[31648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943274, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.189@KRBTEST.COM
I20260620 08:14:34.955638 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.944743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:51485 (local address 127.24.249.188:40053)
0620 08:14:34.944803 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.944805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.944951 (+ 146us) server_negotiation.cc:407] Connection header received
0620 08:14:34.945923 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.945924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.945937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.945970 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.946692 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.947460 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.948571 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.948884 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.948927 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.952429 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.952431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.952431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.952438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.954031 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.954487 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.954488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.954489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.954545 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.954807 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.954808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.954808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.954931 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.954986 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.955457 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.955537 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:14:34.955732 469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.944424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.24.249.131:35889)
0620 08:14:34.944986 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.944993 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.945762 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.946055 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.946056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.946398 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.946585 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.946590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.947570 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.947571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.948468 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.948470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.949016 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.949016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.949076 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.949671 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.949675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.952689 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.954483 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.954484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.954485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.954686 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.954889 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.954889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.954890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.954943 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.955206 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.955207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.955244 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.955314 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.955492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":121,"thread_start_us":82,"threads_started":1}
I20260620 08:14:34.955945 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.944424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.24.249.131:51485)
0620 08:14:34.944896 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.944905 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.945752 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.946048 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.946049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.946451 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.946588 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.946594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.947557 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.947558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.948468 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.948470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.949008 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.949009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.949076 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.949591 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.949595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.952346 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.954114 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.954122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.954125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.954405 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.954605 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.954606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.954606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.954723 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.955011 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.955017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.955184 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.955292 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.955859 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":419,"thread_start_us":91,"threads_started":1}
I20260620 08:14:34.956234 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.944522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:35889 (local address 127.24.249.189:39301)
0620 08:14:34.944569 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.944571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.945049 (+ 478us) server_negotiation.cc:407] Connection header received
0620 08:14:34.945936 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.945938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.945949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.945979 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.946707 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.947489 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.948571 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.948911 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.948948 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.952774 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.952775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.952775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.952783 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.954380 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.954766 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.954768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.954768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.954824 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.955002 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.955002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.955003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.955118 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.955183 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.956049 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.956142 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:14:34.957146 463 heartbeater.cc:347] Connected to a master server at 127.24.249.188:40053
I20260620 08:14:34.957240 463 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.957441 463 heartbeater.cc:511] Master 127.24.249.188:40053 requested a full tablet report, sending...
I20260620 08:14:34.958209 465 heartbeater.cc:347] Connected to a master server at 127.24.249.189:39301
I20260620 08:14:34.958360 465 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.958420 465 heartbeater.cc:511] Master 127.24.249.189:39301 requested a full tablet report, sending...
I20260620 08:14:34.958703 32292 ts_manager.cc:194] Registered new tserver with Master: 23117b06df41450d8c1838e1bb225f1d (127.24.249.131:41293)
I20260620 08:14:34.958827 32479 ts_manager.cc:194] Registered new tserver with Master: 23117b06df41450d8c1838e1bb225f1d (127.24.249.131:41293)
I20260620 08:14:34.960587 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.944423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:53163)
0620 08:14:34.944647 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.944690 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.945747 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.946045 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.946049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.946480 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:34.946588 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.946594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.947311 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.947312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.948106 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.948108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.948519 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.948521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.948658 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.949197 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:34.949204 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:34.953675 (+ 4471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:34.958217 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.958218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.958219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.958439 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.958663 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:34.958664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:34.958664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:34.958718 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:34.959845 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:34.959846 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:34.959885 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.960340 (+ 455us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.960443 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260620 08:14:34.960618 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.944758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:53163 (local address 127.24.249.190:40223)
0620 08:14:34.944802 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.944804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.944814 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:34.945896 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.945898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.945911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.945973 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.946719 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.947204 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.948223 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.948414 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.948515 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.956358 (+ 7843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.956359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.956359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.956366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.958110 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.958526 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.958527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.958527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.958588 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.959601 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.959602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.959602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.959745 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.959795 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.960403 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.960489 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:14:34.960922 464 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40223
I20260620 08:14:34.960968 464 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:34.961028 464 heartbeater.cc:511] Master 127.24.249.190:40223 requested a full tablet report, sending...
I20260620 08:14:34.961262 31689 ts_manager.cc:194] Re-registered known tserver with Master: 23117b06df41450d8c1838e1bb225f1d (127.24.249.131:41293)
I20260620 08:14:34.961547 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.951345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.24.249.188:40053)
0620 08:14:34.951559 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.951561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.951569 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:34.951590 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.951590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.951602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.951670 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.952569 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.954320 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.955666 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.955963 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.956010 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.958586 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.958588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.958588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.958594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.960253 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.960654 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.960656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.960657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.960718 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.960930 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.960931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.960932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.961065 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.961144 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.961357 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.961434 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":41,"threads_started":1}
I20260620 08:14:34.961778 31689 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:53163
I20260620 08:14:34.962760 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.951181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.24.249.190:40223)
0620 08:14:34.952035 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.952037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.952044 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:34.952066 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.952067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.952078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.952849 (+ 771us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.952891 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.953688 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.954794 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.955085 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.955433 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.956736 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.956738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.956738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.956745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.961549 (+ 4804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.961988 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.961989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.961989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.962045 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.962254 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.962256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.962256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.962366 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.962472 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.962571 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.962652 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":65,"threads_started":1}
I20260620 08:14:34.964521 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.951710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.24.249.189:39301)
0620 08:14:34.951749 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.951751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.951977 (+ 226us) server_negotiation.cc:407] Connection header received
0620 08:14:34.952076 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.952077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.952088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.952120 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:34.956459 (+ 4339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.957228 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.958233 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.958548 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.958586 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.960176 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:34.960178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:34.960178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:34.960185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:34.963247 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.963637 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.963638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.963639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.963701 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:34.963924 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:34.963925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:34.963926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:34.964053 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:34.964162 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.964308 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.964416 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:14:34.964648 31689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:34.966329 31689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:34.974500 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.970428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.24.249.130:45567)
0620 08:14:34.970741 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.970747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.970758 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:34.970798 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.970800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.970819 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.971052 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:34.971096 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.971852 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.973305 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.974124 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.974227 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.974340 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.974367 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.974409 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":79,"threads_started":1}
I20260620 08:14:34.974490 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.969855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45567 (local address 127.0.0.1:55114)
0620 08:14:34.969931 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.969941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.969977 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.970934 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.970935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.970937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:34.970981 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.970982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.971947 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.971948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.973204 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.973206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.974218 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.974219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.974285 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.974289 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.974341 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.974387 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 08:14:34.974721 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.970031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:34493 (local address 127.0.0.1:58238)
0620 08:14:34.970185 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.970196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.970242 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.970821 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.970822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.970825 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:34.970916 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.970918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.972218 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.972219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.973384 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.973386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.974335 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.974336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.974416 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.974420 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.974451 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.974489 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260620 08:14:34.975097 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.970109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.24.249.129:34493)
0620 08:14:34.970605 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.970609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.970618 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:34.970653 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.970655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.970672 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.970981 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:34.971038 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.972143 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.973459 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.974241 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.974898 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.974958 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.974982 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.975023 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":111,"threads_started":1}
I20260620 08:14:34.975284 32729 tablet_service.cc:1467] Processing CreateTablet for tablet cc01d8cb926e4003a142a17b9d382fac (DEFAULT_TABLE table=test-table [id=0531b68bddeb442687053b9b5270edae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:34.975622 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc01d8cb926e4003a142a17b9d382fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:34.975754 32592 tablet_service.cc:1467] Processing CreateTablet for tablet cc01d8cb926e4003a142a17b9d382fac (DEFAULT_TABLE table=test-table [id=0531b68bddeb442687053b9b5270edae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:34.976104 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.970267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:41293 (local address 127.0.0.1:58554)
0620 08:14:34.970390 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:34.970401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:34.970459 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:34.970825 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:34.970825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:34.970827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:34.970909 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.970911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.973455 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.973456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.974391 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:34.974393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.975855 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.975856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:34.975921 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.975926 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:34.975995 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 08:14:34.976033 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260620 08:14:34.976130 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc01d8cb926e4003a142a17b9d382fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:34.976138 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:34.970390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.24.249.131:41293)
0620 08:14:34.970563 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:34.970569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:34.970581 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:34.970623 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:34.970626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:34.970643 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:34.970736 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:34.971024 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.972125 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.974483 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:34.975767 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:34.975871 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:34.975924 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:34.976009 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 08:14:34.976053 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260620 08:14:34.976732 397 tablet_service.cc:1467] Processing CreateTablet for tablet cc01d8cb926e4003a142a17b9d382fac (DEFAULT_TABLE table=test-table [id=0531b68bddeb442687053b9b5270edae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:34.977033 397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc01d8cb926e4003a142a17b9d382fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:34.979218 488 tablet_bootstrap.cc:492] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Bootstrap starting.
I20260620 08:14:34.979923 488 tablet_bootstrap.cc:654] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:34.980073 489 tablet_bootstrap.cc:492] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Bootstrap starting.
I20260620 08:14:34.980309 488 log.cc:824] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:34.980856 489 tablet_bootstrap.cc:654] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:34.981164 489 log.cc:824] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:34.981218 488 tablet_bootstrap.cc:492] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: No bootstrap required, opened a new log
I20260620 08:14:34.981276 488 ts_tablet_manager.cc:1397] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:34.981956 489 tablet_bootstrap.cc:492] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: No bootstrap required, opened a new log
I20260620 08:14:34.982030 489 ts_tablet_manager.cc:1397] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:34.982046 491 tablet_bootstrap.cc:492] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Bootstrap starting.
I20260620 08:14:34.982574 491 tablet_bootstrap.cc:654] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:34.982842 491 log.cc:824] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:34.983479 491 tablet_bootstrap.cc:492] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: No bootstrap required, opened a new log
I20260620 08:14:34.983567 491 ts_tablet_manager.cc:1397] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:34.984318 488 raft_consensus.cc:348] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:34.984537 488 raft_consensus.cc:374] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:34.984591 488 raft_consensus.cc:729] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf52a076ee246b6a9b2ac32ffbfcdb4, State: Initialized, Role: FOLLOWER
I20260620 08:14:34.984771 488 consensus_queue.cc:260] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:34.984894 489 raft_consensus.cc:348] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:34.985062 488 ts_tablet_manager.cc:1428] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:14:34.985060 489 raft_consensus.cc:374] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:34.985103 489 raft_consensus.cc:729] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df02488ec6943258258b470d8991b9b, State: Initialized, Role: FOLLOWER
I20260620 08:14:34.985186 32659 heartbeater.cc:503] Master 127.24.249.190:40223 was elected leader, sending a full tablet report...
I20260620 08:14:34.985260 489 consensus_queue.cc:260] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:34.985502 489 ts_tablet_manager.cc:1428] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 08:14:34.985770 328 heartbeater.cc:503] Master 127.24.249.190:40223 was elected leader, sending a full tablet report...
I20260620 08:14:34.986982 491 raft_consensus.cc:348] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:34.987130 491 raft_consensus.cc:374] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:34.987174 491 raft_consensus.cc:729] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23117b06df41450d8c1838e1bb225f1d, State: Initialized, Role: FOLLOWER
I20260620 08:14:34.987295 491 consensus_queue.cc:260] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:34.987569 491 ts_tablet_manager.cc:1428] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 08:14:34.987633 464 heartbeater.cc:503] Master 127.24.249.190:40223 was elected leader, sending a full tablet report...
W20260620 08:14:35.054643 330 tablet.cc:2369] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:35.057444 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.054599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.0.0.1:57788)
0620 08:14:35.054682 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:35.054696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:35.054791 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:35.054965 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:35.054966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:35.054967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:35.055018 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:35.055020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.055774 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.055776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:35.056520 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:35.056522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.057151 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.057153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:35.057267 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.057272 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:35.057336 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:14:35.057380 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 08:14:35.057477 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.054745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.24.249.188:40053)
0620 08:14:35.054795 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:35.054797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:35.054809 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:35.054855 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:35.054856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:35.054879 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:35.054955 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:35.055145 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.055675 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.056618 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.057036 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.057107 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.057163 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:35.057372 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 08:14:35.057425 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:14:35.058055 32487 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 08:14:35.059407 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1502f2ee3724eb980ad47e8a86d494b. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:35.059548 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: This master's current role is: LEARNER
I20260620 08:14:35.065399 496 raft_consensus.cc:482] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:35.065514 496 raft_consensus.cc:504] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:35.066138 496 leader_election.cc:288] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567), 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493)
I20260620 08:14:35.069326 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.066471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:52209 (local address 127.24.249.129:34493)
0620 08:14:35.066532 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:35.066534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:35.066546 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:35.066575 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:35.066576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:35.066586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:35.066645 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:35.066846 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.067418 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.068445 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.068975 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.069044 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.069053 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:35.069209 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 08:14:35.069245 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:14:35.069344 467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.066395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:34493 (local address 127.24.249.131:52209)
0620 08:14:35.066454 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:35.066467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:35.066510 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:35.066665 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:35.066665 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:35.066667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:35.066741 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:35.066744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.067499 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.067500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:35.068358 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:35.068361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.069057 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.069058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:35.069136 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.069140 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:35.069200 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:14:35.069242 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 08:14:35.069363 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.066539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:49151 (local address 127.24.249.130:45567)
0620 08:14:35.066580 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:35.066581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:35.066590 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:35.066665 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:35.066665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:35.066675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:35.066785 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:35.066985 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.067442 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.068473 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.068961 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.069034 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.069043 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:35.069254 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 08:14:35.069293 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:35.069422 468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.066479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45567 (local address 127.24.249.131:49151)
0620 08:14:35.066528 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:35.066539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:35.066578 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:35.066846 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:35.066847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:35.066848 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:35.066905 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:35.066906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.067556 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.067557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:35.068368 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:35.068370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.069076 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.069077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:35.069164 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.069169 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:35.069247 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 08:14:35.069288 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:14:35.069748 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc01d8cb926e4003a142a17b9d382fac" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df02488ec6943258258b470d8991b9b" is_pre_election: true
I20260620 08:14:35.069733 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc01d8cb926e4003a142a17b9d382fac" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" is_pre_election: true
I20260620 08:14:35.069921 32749 raft_consensus.cc:2454] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 0.
I20260620 08:14:35.069921 32612 raft_consensus.cc:2454] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 0.
I20260620 08:14:35.070178 353 leader_election.cc:302] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23117b06df41450d8c1838e1bb225f1d, 7df02488ec6943258258b470d8991b9b; no voters:
I20260620 08:14:35.070367 496 raft_consensus.cc:2781] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:35.070441 496 raft_consensus.cc:482] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:35.070477 496 raft_consensus.cc:3037] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:35.071209 496 raft_consensus.cc:504] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:35.071378 496 leader_election.cc:288] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 election: Requested vote from peers 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567), 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493)
I20260620 08:14:35.071542 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc01d8cb926e4003a142a17b9d382fac" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4"
I20260620 08:14:35.071599 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc01d8cb926e4003a142a17b9d382fac" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df02488ec6943258258b470d8991b9b"
I20260620 08:14:35.071660 32612 raft_consensus.cc:3037] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:35.071669 32749 raft_consensus.cc:3037] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:35.072636 32612 raft_consensus.cc:2454] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 1.
I20260620 08:14:35.072671 32749 raft_consensus.cc:2454] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 1.
I20260620 08:14:35.072813 353 leader_election.cc:302] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23117b06df41450d8c1838e1bb225f1d, 7df02488ec6943258258b470d8991b9b; no voters:
I20260620 08:14:35.072960 496 raft_consensus.cc:2781] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:35.073127 496 raft_consensus.cc:686] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 1 LEADER]: Becoming Leader. State: Replica: 23117b06df41450d8c1838e1bb225f1d, State: Running, Role: LEADER
I20260620 08:14:35.073215 496 consensus_queue.cc:237] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } }
I20260620 08:14:35.074035 31689 catalog_manager.cc:5495] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 23117b06df41450d8c1838e1bb225f1d (127.24.249.131). New cstate: current_term: 1 leader_uuid: "23117b06df41450d8c1838e1bb225f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } health_report { overall_health: HEALTHY } } }
I20260620 08:14:35.074748 32252 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: NON_VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: true } } }
I20260620 08:14:35.074857 32252 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:35.075968 32252 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: attempting to promote NON_VOTER 529cd900b5224f8cad12fd4ba462f268 to VOTER
I20260620 08:14:35.076112 32252 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } }
I20260620 08:14:35.076731 32487 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 LEARNER]: Refusing update from remote peer e1502f2ee3724eb980ad47e8a86d494b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 08:14:35.076749 32302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Refusing update from remote peer e1502f2ee3724eb980ad47e8a86d494b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 08:14:35.077019 32253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Connected to new peer: Peer: permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260620 08:14:35.077113 502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [LEADER]: Connected to new peer: Peer: permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 08:14:35.077698 32253 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.078537 32253 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.078620 32253 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:35.078641 502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.078647 504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.078766 504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:35.078796 502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:35.078768 32487 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.079707 32302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.079735 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.079841 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:14:35.080581 506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1502f2ee3724eb980ad47e8a86d494b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:35.080690 506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:14:35.096496 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.094363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.24.249.131:41293)
0620 08:14:35.094404 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:35.094406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:35.094416 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:35.094467 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:35.094468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:35.094485 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:35.094549 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:35.094662 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.095142 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.095866 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.096047 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.096100 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.096237 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:35.096332 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:35.096390 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:35.096422 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 08:14:35.096438 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:14:35.098356 32749 raft_consensus.cc:1261] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Refusing update from remote peer 23117b06df41450d8c1838e1bb225f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:35.098670 32612 raft_consensus.cc:1261] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 1 FOLLOWER]: Refusing update from remote peer 23117b06df41450d8c1838e1bb225f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:35.098701 496 consensus_queue.cc:1035] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:35.098970 496 consensus_queue.cc:1035] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:35.102846 510 mvcc.cc:205] Tried to move back new op lower bound from 7298839654799876096 to 7298839654700482560. Current Snapshot: MvccSnapshot[applied={T|T < 7298839654799876096}]
I20260620 08:14:35.103288 509 mvcc.cc:205] Tried to move back new op lower bound from 7298839654799876096 to 7298839654700482560. Current Snapshot: MvccSnapshot[applied={T|T < 7298839654799876096 or (T in {7298839654799876096})}]
I20260620 08:14:35.105477 31689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:35.105576 31689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:35.106949 31689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1502f2ee3724eb980ad47e8a86d494b: Sending DeleteTablet for 3 replicas of tablet cc01d8cb926e4003a142a17b9d382fac
I20260620 08:14:35.107244 32729 tablet_service.cc:1514] Processing DeleteTablet for tablet cc01d8cb926e4003a142a17b9d382fac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:35 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:55114
I20260620 08:14:35.107299 397 tablet_service.cc:1514] Processing DeleteTablet for tablet cc01d8cb926e4003a142a17b9d382fac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:35 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:58554
I20260620 08:14:35.107530 32592 tablet_service.cc:1514] Processing DeleteTablet for tablet cc01d8cb926e4003a142a17b9d382fac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:35 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:58238
I20260620 08:14:35.108824 520 tablet_replica.cc:330] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: stopping tablet replica
I20260620 08:14:35.108937 520 raft_consensus.cc:2229] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:35.109042 520 raft_consensus.cc:2258] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:35.109191 518 tablet_replica.cc:330] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: stopping tablet replica
I20260620 08:14:35.109293 518 raft_consensus.cc:2229] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:35.109404 518 raft_consensus.cc:2258] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:35.109508 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 31661
I20260620 08:14:35.109820 520 ts_tablet_manager.cc:1905] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:35.111891 520 ts_tablet_manager.cc:1918] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:35.111971 520 log.cc:1196] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/cc01d8cb926e4003a142a17b9d382fac
I20260620 08:14:35.112252 520 ts_tablet_manager.cc:1939] T cc01d8cb926e4003a142a17b9d382fac P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Deleting consensus metadata
I20260620 08:14:35.112946 518 ts_tablet_manager.cc:1905] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:35.114331 518 ts_tablet_manager.cc:1918] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:35.114419 518 log.cc:1196] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/cc01d8cb926e4003a142a17b9d382fac
I20260620 08:14:35.114679 518 ts_tablet_manager.cc:1939] T cc01d8cb926e4003a142a17b9d382fac P 23117b06df41450d8c1838e1bb225f1d: Deleting consensus metadata
I20260620 08:14:35.115103 517 tablet_replica.cc:330] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: stopping tablet replica
I20260620 08:14:35.115229 517 raft_consensus.cc:2229] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:35.115345 517 raft_consensus.cc:2258] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:35.115734 517 ts_tablet_manager.cc:1905] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:35.116926 517 ts_tablet_manager.cc:1918] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:35.117015 517 log.cc:1196] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/cc01d8cb926e4003a142a17b9d382fac
I20260620 08:14:35.117249 517 ts_tablet_manager.cc:1939] T cc01d8cb926e4003a142a17b9d382fac P 7df02488ec6943258258b470d8991b9b: Deleting consensus metadata
W20260620 08:14:35.117642 353 connection.cc:535] server connection from 127.0.0.1:58554 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 08:14:35.117645 32545 connection.cc:535] server connection from 127.0.0.1:58238 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:14:35.137773 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.119418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.24.249.188:40053)
0620 08:14:35.119463 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:35.119465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:35.119576 (+ 111us) server_negotiation.cc:407] Connection header received
0620 08:14:35.119657 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:35.119658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:35.119669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:35.119721 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:35.120492 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.121243 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.122297 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.122557 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.122604 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.134660 (+ 12056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:35.134663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:35.134663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:35.134672 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:35.136450 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:35.136888 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:35.136890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:35.136890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:35.136945 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:35.137159 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:35.137160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:35.137160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:35.137298 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:35.137375 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:35.137583 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 08:14:35.137667 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:14:35.138509 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.119287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.24.249.189:39301)
0620 08:14:35.119331 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:35.119333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:35.119511 (+ 178us) server_negotiation.cc:407] Connection header received
0620 08:14:35.119596 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:35.119597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:35.119609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:35.119684 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:35.120444 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.121029 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.121828 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.122084 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.122189 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.134792 (+ 12603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:35.134794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:35.134795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:35.134803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:35.136804 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:35.137254 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:35.137255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:35.137256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:35.137316 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:35.137525 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:35.137526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:35.137526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:35.137677 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:35.137755 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:35.138329 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 08:14:35.138416 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:14:35.138609 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:35.107718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.24.249.188:40053)
0620 08:14:35.107762 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:35.107764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:35.107774 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:35.107835 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:35.107836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:35.107847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:35.107906 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:35.108596 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.109360 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.111453 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:35.111760 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:35.111807 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:35.134779 (+ 22972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:35.134780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:35.134781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:35.134789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:35.137291 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:35.137758 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:35.137760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:35.137760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:35.137813 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:35.138038 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:35.138038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:35.138039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:35.138160 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:35.138242 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:35.138444 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 08:14:35.138521 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:14:36.102720 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.102365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.130:50211)
0620 08:14:36.102519 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.102662 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
W20260620 08:14:36.103016 328 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:40223 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.103503 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.103029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:40533)
0620 08:14:36.103386 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.103457 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":55,"threads_started":1}
W20260620 08:14:36.103715 464 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:40223 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.103937 530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.103497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.129:53065)
0620 08:14:36.103830 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.103896 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":67,"threads_started":1}
W20260620 08:14:36.104149 32659 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:40223 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.607105 531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1502f2ee3724eb980ad47e8a86d494b)
I20260620 08:14:36.607268 531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } }
I20260620 08:14:36.607538 531 leader_election.cc:288] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1502f2ee3724eb980ad47e8a86d494b (127.24.249.190:40223), 684f14a55dc6408d9d7e9dd2048362ce (127.24.249.189:39301)
I20260620 08:14:36.608029 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.607805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53916)
0620 08:14:36.607934 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.607989 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20260620 08:14:36.608163 32465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.608641 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.608513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53918)
0620 08:14:36.608586 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.608613 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 08:14:36.608749 32465 leader_election.cc:334] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1502f2ee3724eb980ad47e8a86d494b (127.24.249.190:40223): Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.609344 536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1502f2ee3724eb980ad47e8a86d494b)
I20260620 08:14:36.609494 536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } }
I20260620 08:14:36.609789 536 leader_election.cc:288] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1502f2ee3724eb980ad47e8a86d494b (127.24.249.190:40223), 529cd900b5224f8cad12fd4ba462f268 (127.24.249.188:40053)
I20260620 08:14:36.610116 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.609888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53928)
0620 08:14:36.609992 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.610051 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
W20260620 08:14:36.610216 32278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.610623 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.610522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53934)
0620 08:14:36.610571 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.610601 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:14:36.610721 32278 leader_election.cc:334] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1502f2ee3724eb980ad47e8a86d494b (127.24.249.190:40223): Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.611608 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.607882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.24.249.189:39301)
0620 08:14:36.607988 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:36.607990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:36.608002 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:36.608034 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:36.608035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:36.608045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:36.608105 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:36.608293 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.608957 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.610795 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.611302 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.611359 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.611366 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:36.611522 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 08:14:36.611554 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
I20260620 08:14:36.611615 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.607795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:39301 (local address 127.0.0.1:39980)
0620 08:14:36.607902 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.607915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:36.607979 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:36.608149 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:36.608151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:36.608153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:36.608214 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.608216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.609124 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.609126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.610648 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.610652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.611378 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.611379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.611440 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.611444 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:36.611504 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:14:36.611544 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260620 08:14:36.611847 32302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "529cd900b5224f8cad12fd4ba462f268" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "684f14a55dc6408d9d7e9dd2048362ce" is_pre_election: true
I20260620 08:14:36.611955 32302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529cd900b5224f8cad12fd4ba462f268 in term 1.
I20260620 08:14:36.612125 32466 leader_election.cc:302] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 529cd900b5224f8cad12fd4ba462f268, 684f14a55dc6408d9d7e9dd2048362ce; no voters: e1502f2ee3724eb980ad47e8a86d494b
I20260620 08:14:36.612287 531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 08:14:36.612360 531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1502f2ee3724eb980ad47e8a86d494b)
I20260620 08:14:36.612381 531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:14:36.613413 531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } }
I20260620 08:14:36.613564 531 leader_election.cc:288] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [CANDIDATE]: Term 2 election: Requested vote from peers e1502f2ee3724eb980ad47e8a86d494b (127.24.249.190:40223), 684f14a55dc6408d9d7e9dd2048362ce (127.24.249.189:39301)
I20260620 08:14:36.614010 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.613688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53946)
0620 08:14:36.613950 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.613977 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260620 08:14:36.614156 32302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "529cd900b5224f8cad12fd4ba462f268" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "684f14a55dc6408d9d7e9dd2048362ce"
I20260620 08:14:36.614212 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.610022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.24.249.188:40053)
0620 08:14:36.610772 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:36.610774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:36.610780 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:14:36.610800 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:36.610801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:36.610810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:36.610877 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:36.611032 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.611618 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.612939 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.613492 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.613574 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.613584 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:36.614113 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 08:14:36.614160 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":76,"threads_started":1}
I20260620 08:14:36.614177 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.609931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:40053 (local address 127.0.0.1:57814)
0620 08:14:36.610077 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.610088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:36.610138 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:36.610886 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:36.610887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:36.610889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:36.610958 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.610960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.611757 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.611758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.612840 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.612843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.613741 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.613742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.613945 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.613951 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:14:36.614058 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:14:36.614098 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":29,"thread_start_us":56,"threads_started":1}
I20260620 08:14:36.614238 32302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:14:36.614514 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "684f14a55dc6408d9d7e9dd2048362ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "529cd900b5224f8cad12fd4ba462f268" is_pre_election: true
I20260620 08:14:36.614640 32487 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 684f14a55dc6408d9d7e9dd2048362ce in current term 2: Already voted for candidate 529cd900b5224f8cad12fd4ba462f268 in this term.
I20260620 08:14:36.614837 32276 leader_election.cc:302] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 684f14a55dc6408d9d7e9dd2048362ce; no voters: 529cd900b5224f8cad12fd4ba462f268, e1502f2ee3724eb980ad47e8a86d494b
I20260620 08:14:36.615072 32302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529cd900b5224f8cad12fd4ba462f268 in term 2.
I20260620 08:14:36.615155 536 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:14:36.615249 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.614836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53972)
0620 08:14:36.615193 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.615223 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340}
I20260620 08:14:36.615255 32466 leader_election.cc:302] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 529cd900b5224f8cad12fd4ba462f268, 684f14a55dc6408d9d7e9dd2048362ce; no voters:
I20260620 08:14:36.615350 531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 2 FOLLOWER]: Leader election won for term 2
W20260620 08:14:36.615355 32465 leader_election.cc:334] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1502f2ee3724eb980ad47e8a86d494b (127.24.249.190:40223): Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
I20260620 08:14:36.615484 531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [term 2 LEADER]: Becoming Leader. State: Replica: 529cd900b5224f8cad12fd4ba462f268, State: Running, Role: LEADER
I20260620 08:14:36.615577 531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } }
I20260620 08:14:36.615897 541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529cd900b5224f8cad12fd4ba462f268. Latest consensus state: current_term: 2 leader_uuid: "529cd900b5224f8cad12fd4ba462f268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:36.615990 541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:36.616143 543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:36.616559 543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:36.616616 543 catalog_manager.cc:1259] Loaded cluster ID: 690e28b1718b448f95c84776431423b2
I20260620 08:14:36.616669 543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:36.616837 543 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:36.616902 543 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Loaded TSK: 0
I20260620 08:14:36.617043 543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:14:36.672560 32479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:36.674345 32479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:36.675439 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.675317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53980)
0620 08:14:36.675374 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.675409 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 08:14:36.675431 32302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [term 2 FOLLOWER]: Refusing update from remote peer 529cd900b5224f8cad12fd4ba462f268: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 08:14:36.675809 531 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 08:14:36.675884 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.675797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.0.0.1:53990)
0620 08:14:36.675829 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.675858 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 08:14:36.675976 32465 consensus_peers.cc:487] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 -> Peer e1502f2ee3724eb980ad47e8a86d494b (127.24.249.190:40223): Couldn't send request to peer e1502f2ee3724eb980ad47e8a86d494b. Status: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 08:14:36.676465 536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529cd900b5224f8cad12fd4ba462f268. Latest consensus state: current_term: 2 leader_uuid: "529cd900b5224f8cad12fd4ba462f268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:36.676570 536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:14:36.676892 544 mvcc.cc:205] Tried to move back new op lower bound from 7298839661260382208 to 7298839661017878528. Current Snapshot: MvccSnapshot[applied={T|T < 7298839661260382208}]
I20260620 08:14:36.676986 541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "529cd900b5224f8cad12fd4ba462f268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:36.677102 541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:36.677105 536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "529cd900b5224f8cad12fd4ba462f268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e1502f2ee3724eb980ad47e8a86d494b" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40223 } } peers { permanent_uuid: "684f14a55dc6408d9d7e9dd2048362ce" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 39301 } attrs { promote: false } } peers { permanent_uuid: "529cd900b5224f8cad12fd4ba462f268" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 40053 } attrs { promote: false } } }
I20260620 08:14:36.677181 536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684f14a55dc6408d9d7e9dd2048362ce [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:14:36.677232 545 mvcc.cc:205] Tried to move back new op lower bound from 7298839661260382208 to 7298839661017878528. Current Snapshot: MvccSnapshot[applied={T|T < 7298839661260382208 or (T in {7298839661260382208})}]
I20260620 08:14:36.682169 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.678909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:41293 (local address 127.0.0.1:58578)
0620 08:14:36.678970 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.678981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:36.679024 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:36.679257 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:36.679258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:36.679260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:36.679312 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.679314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.680323 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.680324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.681118 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.681120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.681963 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.681964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.682023 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.682028 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:36.682056 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 08:14:36.682097 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260620 08:14:36.682273 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.678972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45567 (local address 127.0.0.1:55116)
0620 08:14:36.679136 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.679149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:36.679217 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:36.679401 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:36.679403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:36.679405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:36.679452 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.679453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.680211 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.680212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.681319 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.681321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.682012 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.682013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.682074 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.682076 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:36.682165 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:14:36.682204 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 08:14:36.682372 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.679089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.24.249.130:45567)
0620 08:14:36.679258 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:36.679260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:36.679268 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:36.679296 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:36.679297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:36.679307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:36.679519 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:36.679558 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.680139 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.681412 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.681941 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.682015 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.682025 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:36.682173 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 08:14:36.682213 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":52,"threads_started":1}
I20260620 08:14:36.682432 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.678977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.24.249.131:41293)
0620 08:14:36.679122 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:36.679124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:36.679131 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:36.679158 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:36.679159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:36.679170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:36.679375 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:36.679445 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.680233 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.681243 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.681853 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.682292 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.682300 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:36.682320 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 08:14:36.682362 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260620 08:14:36.682611 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.678991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:34493 (local address 127.0.0.1:58242)
0620 08:14:36.679348 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.679357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:36.679400 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:36.679583 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:36.679583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:36.679599 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:36.679634 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.679635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.680492 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.680493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.681552 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:36.681554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.682397 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.682398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:36.682458 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.682462 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:36.682508 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:14:36.682552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":62,"threads_started":1}
I20260620 08:14:36.682603 32729 tablet_service.cc:1467] Processing CreateTablet for tablet 78c197e24e364103a8d31f35bbd611ff (DEFAULT_TABLE table=test-table [id=addb410a96a6481c901175d611473627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:36.682737 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78c197e24e364103a8d31f35bbd611ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:36.682802 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.679133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.24.249.129:34493)
0620 08:14:36.679239 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:36.679242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:36.679449 (+ 207us) server_negotiation.cc:407] Connection header received
0620 08:14:36.679475 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:36.679476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:36.679487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:36.679542 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:36.679717 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.680415 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.681629 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.682228 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.682293 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.682300 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:36.682697 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 08:14:36.682743 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
I20260620 08:14:36.682972 32592 tablet_service.cc:1467] Processing CreateTablet for tablet 78c197e24e364103a8d31f35bbd611ff (DEFAULT_TABLE table=test-table [id=addb410a96a6481c901175d611473627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:36.683068 32592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78c197e24e364103a8d31f35bbd611ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:36.683070 397 tablet_service.cc:1467] Processing CreateTablet for tablet 78c197e24e364103a8d31f35bbd611ff (DEFAULT_TABLE table=test-table [id=addb410a96a6481c901175d611473627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:36.683180 397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78c197e24e364103a8d31f35bbd611ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:36.684298 554 tablet_bootstrap.cc:492] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: Bootstrap starting.
I20260620 08:14:36.684446 555 tablet_bootstrap.cc:492] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Bootstrap starting.
I20260620 08:14:36.684696 554 tablet_bootstrap.cc:654] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:36.684875 556 tablet_bootstrap.cc:492] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Bootstrap starting.
I20260620 08:14:36.684985 555 tablet_bootstrap.cc:654] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:36.685278 556 tablet_bootstrap.cc:654] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:36.685580 555 tablet_bootstrap.cc:492] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: No bootstrap required, opened a new log
I20260620 08:14:36.685647 555 ts_tablet_manager.cc:1397] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:36.685760 555 raft_consensus.cc:348] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.685837 555 raft_consensus.cc:374] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:36.685845 556 tablet_bootstrap.cc:492] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4: No bootstrap required, opened a new log
I20260620 08:14:36.685845 554 tablet_bootstrap.cc:492] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: No bootstrap required, opened a new log
I20260620 08:14:36.685889 555 raft_consensus.cc:729] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23117b06df41450d8c1838e1bb225f1d, State: Initialized, Role: FOLLOWER
I20260620 08:14:36.685913 556 ts_tablet_manager.cc:1397] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:36.685913 554 ts_tablet_manager.cc:1397] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:36.685959 555 consensus_queue.cc:260] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.686046 554 raft_consensus.cc:348] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.686065 556 raft_consensus.cc:348] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.686117 554 raft_consensus.cc:374] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:36.686115 555 ts_tablet_manager.cc:1428] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:36.686143 554 raft_consensus.cc:729] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df02488ec6943258258b470d8991b9b, State: Initialized, Role: FOLLOWER
I20260620 08:14:36.686146 556 raft_consensus.cc:374] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:36.686168 556 raft_consensus.cc:729] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf52a076ee246b6a9b2ac32ffbfcdb4, State: Initialized, Role: FOLLOWER
I20260620 08:14:36.686187 554 consensus_queue.cc:260] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.686216 556 consensus_queue.cc:260] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.686350 554 ts_tablet_manager.cc:1428] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:36.686379 556 ts_tablet_manager.cc:1428] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:36.686972 463 heartbeater.cc:503] Master 127.24.249.188:40053 was elected leader, sending a full tablet report...
I20260620 08:14:36.687283 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.686450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:49913)
0620 08:14:36.687204 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.687244 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":76,"threads_started":1}
I20260620 08:14:36.687569 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.687248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.130:56279)
0620 08:14:36.687466 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.687503 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
I20260620 08:14:36.687697 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.687503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:52023)
0620 08:14:36.687628 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.687665 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20260620 08:14:36.687996 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.687078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.129:45385)
0620 08:14:36.687906 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.687935 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":72,"threads_started":1}
I20260620 08:14:36.688162 327 heartbeater.cc:503] Master 127.24.249.188:40053 was elected leader, sending a full tablet report...
I20260620 08:14:36.688365 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.688272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.130:46689)
0620 08:14:36.688313 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.688338 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 08:14:36.688355 560 raft_consensus.cc:482] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:36.688474 560 raft_consensus.cc:504] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.688588 32658 heartbeater.cc:503] Master 127.24.249.188:40053 was elected leader, sending a full tablet report...
I20260620 08:14:36.688771 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.688693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.129:39817)
0620 08:14:36.688722 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.688746 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 08:14:36.688759 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c197e24e364103a8d31f35bbd611ff" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df02488ec6943258258b470d8991b9b" is_pre_election: true
I20260620 08:14:36.688848 32749 raft_consensus.cc:2454] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 0.
I20260620 08:14:36.688783 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c197e24e364103a8d31f35bbd611ff" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" is_pre_election: true
I20260620 08:14:36.688987 32612 raft_consensus.cc:2454] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 0.
I20260620 08:14:36.688982 353 leader_election.cc:302] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23117b06df41450d8c1838e1bb225f1d, 7df02488ec6943258258b470d8991b9b; no voters:
I20260620 08:14:36.689136 567 raft_consensus.cc:2781] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:36.689198 567 raft_consensus.cc:482] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:36.689220 567 raft_consensus.cc:3037] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:36.689296 560 leader_election.cc:288] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493), 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567)
I20260620 08:14:36.690058 567 raft_consensus.cc:504] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.690196 567 leader_election.cc:288] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 election: Requested vote from peers 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493), 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567)
I20260620 08:14:36.690328 32612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c197e24e364103a8d31f35bbd611ff" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4"
I20260620 08:14:36.690332 32749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78c197e24e364103a8d31f35bbd611ff" candidate_uuid: "23117b06df41450d8c1838e1bb225f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df02488ec6943258258b470d8991b9b"
I20260620 08:14:36.690393 32612 raft_consensus.cc:3037] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:36.690402 32749 raft_consensus.cc:3037] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:36.691126 32612 raft_consensus.cc:2454] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 1.
I20260620 08:14:36.691164 32749 raft_consensus.cc:2454] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23117b06df41450d8c1838e1bb225f1d in term 1.
I20260620 08:14:36.691298 351 leader_election.cc:302] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bf52a076ee246b6a9b2ac32ffbfcdb4, 23117b06df41450d8c1838e1bb225f1d; no voters:
I20260620 08:14:36.691380 567 raft_consensus.cc:2781] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:36.691461 567 raft_consensus.cc:686] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 1 LEADER]: Becoming Leader. State: Replica: 23117b06df41450d8c1838e1bb225f1d, State: Running, Role: LEADER
I20260620 08:14:36.691511 567 consensus_queue.cc:237] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } }
I20260620 08:14:36.692111 32477 catalog_manager.cc:5495] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 23117b06df41450d8c1838e1bb225f1d (127.24.249.131). New cstate: current_term: 1 leader_uuid: "23117b06df41450d8c1838e1bb225f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23117b06df41450d8c1838e1bb225f1d" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:36.692765 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.692655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:35039)
0620 08:14:36.692695 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.692732 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 08:14:36.696038 466 tablet.cc:2369] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:36.698668 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.696241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.24.249.131:41293)
0620 08:14:36.696284 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:36.696286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:36.696344 (+ 58us) server_negotiation.cc:407] Connection header received
0620 08:14:36.696395 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:36.696396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:36.696405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:36.696461 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:36.696548 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.697021 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.697852 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:36.698204 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:36.698275 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:36.698440 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:36.698471 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:36.698510 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:36.698566 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 08:14:36.698588 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:36.699524 32749 raft_consensus.cc:1261] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Refusing update from remote peer 23117b06df41450d8c1838e1bb225f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:36.699524 32612 raft_consensus.cc:1261] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4 [term 1 FOLLOWER]: Refusing update from remote peer 23117b06df41450d8c1838e1bb225f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:36.699836 567 consensus_queue.cc:1035] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df02488ec6943258258b470d8991b9b" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:36.699927 560 consensus_queue.cc:1035] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf52a076ee246b6a9b2ac32ffbfcdb4" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:36.700736 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.700565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.130:56033)
0620 08:14:36.700599 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.700632 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 08:14:36.701953 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.701605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.129:40831)
0620 08:14:36.701885 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.701919 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260620 08:14:36.702334 570 mvcc.cc:205] Tried to move back new op lower bound from 7298839661359427584 to 7298839661328785408. Current Snapshot: MvccSnapshot[applied={T|T < 7298839661359427584}]
I20260620 08:14:36.702536 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.702161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.130:43787)
0620 08:14:36.702485 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.702513 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
I20260620 08:14:36.702343 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.702241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:43697)
0620 08:14:36.702284 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.702312 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 08:14:36.702909 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.702830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.131:36017)
0620 08:14:36.702862 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.702885 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 08:14:36.703109 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.703012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40223 (local address 127.24.249.129:40557)
0620 08:14:36.703044 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.703078 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 08:14:36.704932 32478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:36.705013 32478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:36.706202 32478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 529cd900b5224f8cad12fd4ba462f268: Sending DeleteTablet for 3 replicas of tablet 78c197e24e364103a8d31f35bbd611ff
I20260620 08:14:36.706503 397 tablet_service.cc:1514] Processing DeleteTablet for tablet 78c197e24e364103a8d31f35bbd611ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:36 UTC) from {username='oryx', principal='oryx/127.24.249.188@KRBTEST.COM'} at 127.0.0.1:58578
I20260620 08:14:36.706502 32729 tablet_service.cc:1514] Processing DeleteTablet for tablet 78c197e24e364103a8d31f35bbd611ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:36 UTC) from {username='oryx', principal='oryx/127.24.249.188@KRBTEST.COM'} at 127.0.0.1:55116
I20260620 08:14:36.706697 578 tablet_replica.cc:330] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: stopping tablet replica
I20260620 08:14:36.706758 578 raft_consensus.cc:2229] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:36.706755 579 tablet_replica.cc:330] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: stopping tablet replica
I20260620 08:14:36.706750 32592 tablet_service.cc:1514] Processing DeleteTablet for tablet 78c197e24e364103a8d31f35bbd611ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:36 UTC) from {username='oryx', principal='oryx/127.24.249.188@KRBTEST.COM'} at 127.0.0.1:58242
I20260620 08:14:36.706801 578 raft_consensus.cc:2258] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:36.706800 579 raft_consensus.cc:2229] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:36.706863 579 raft_consensus.cc:2258] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:36.707044 580 tablet_replica.cc:330] T 78c197e24e364103a8d31f35bbd611ff P 0bf52a076ee246b6a9b2ac32ffbfcdb4: stopping tablet replica
I20260620 08:14:36.707044 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 32443
I20260620 08:14:36.707157 579 ts_tablet_manager.cc:1905] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:36.707679 578 ts_tablet_manager.cc:1905] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:36.709538 579 ts_tablet_manager.cc:1918] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:36.709615 579 log.cc:1196] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/78c197e24e364103a8d31f35bbd611ff
I20260620 08:14:36.709547 578 ts_tablet_manager.cc:1918] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:36.709697 578 log.cc:1196] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/78c197e24e364103a8d31f35bbd611ff
I20260620 08:14:36.709957 579 ts_tablet_manager.cc:1939] T 78c197e24e364103a8d31f35bbd611ff P 23117b06df41450d8c1838e1bb225f1d: Deleting consensus metadata
I20260620 08:14:36.709970 578 ts_tablet_manager.cc:1939] T 78c197e24e364103a8d31f35bbd611ff P 7df02488ec6943258258b470d8991b9b: Deleting consensus metadata
I20260620 08:14:36.710310 32466 catalog_manager.cc:4841] TS 7df02488ec6943258258b470d8991b9b (127.24.249.130:45567): tablet 78c197e24e364103a8d31f35bbd611ff (table test-table [id=addb410a96a6481c901175d611473627]) successfully deleted
I20260620 08:14:36.710345 32464 catalog_manager.cc:4841] TS 23117b06df41450d8c1838e1bb225f1d (127.24.249.131:41293): tablet 78c197e24e364103a8d31f35bbd611ff (table test-table [id=addb410a96a6481c901175d611473627]) successfully deleted
I20260620 08:14:36.713654 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:36.713471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:34493 (local address 127.0.0.1:58246)
0620 08:14:36.713543 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:36.713622 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:34493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 08:14:36.713690 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 32666
W20260620 08:14:36.713840 32464 catalog_manager.cc:4584] TS 0bf52a076ee246b6a9b2ac32ffbfcdb4 (127.24.249.129:34493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78c197e24e364103a8d31f35bbd611ff: Network error: Client connection negotiation failed: client connection to 127.24.249.129:34493: connect: Connection refused (error 111)
I20260620 08:14:36.719246 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 334
I20260620 08:14:36.725450 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 32157
I20260620 08:14:36.730361 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 32346
2026-06-20T08:14:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4530 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:36 dist-test-slave-65w8 krb5kdc[583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:36 dist-test-slave-65w8 krb5kdc[583](info): set up 2 sockets
Jun 20 08:14:36 dist-test-slave-65w8 krb5kdc[583](info): commencing operation
krb5kdc: starting...
W20260620 08:14:37.806641 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:37 dist-test-slave-65w8 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab.
Entry for principal oryx/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab.
I20260620 08:14:37.935595 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:38875
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab
--principal=oryx/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:38875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:38.018461 596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:38.018743 596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:38.018786 596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:38.019982 596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:38.020058 596 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 08:14:38.020081 596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:38.020103 596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:38.020128 596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:38.020159 596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:38.021798 596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:38875
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.190.keytab
--principal=oryx/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:38875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:38.022141 596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:38.022395 596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:38.027556 601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:38.027593 596 server_base.cc:1034] running on GCE node
W20260620 08:14:38.027570 604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:38.027561 602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:38.028030 596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:38.028370 596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:38.029537 596 hybrid_clock.cc:648] HybridClock initialized: now 1781943278029522 us; error 31 us; skew 500 ppm
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:38.032321 596 init.cc:377] Logged in from keytab as oryx/127.24.249.190@KRBTEST.COM (short username oryx)
I20260620 08:14:38.032823 596 webserver.cc:460] Webserver started at http://127.24.249.190:41847/ using document root <none> and password file <none>
I20260620 08:14:38.033079 596 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:38.033151 596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:38.033299 596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:38.034468 596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "cc25077815224057ac32edd836740f6c"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "0061e5fec5f16204525105e7e41db012"
server_key_iv: "8021a5e7a40d122341ae3b889e243cfc"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.034850 596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "cc25077815224057ac32edd836740f6c"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "0061e5fec5f16204525105e7e41db012"
server_key_iv: "8021a5e7a40d122341ae3b889e243cfc"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.036296 596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:38.037247 611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.037643 596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:38.037818 596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "cc25077815224057ac32edd836740f6c"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "0061e5fec5f16204525105e7e41db012"
server_key_iv: "8021a5e7a40d122341ae3b889e243cfc"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.037961 596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:38.060298 596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:38.062741 596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:38.062922 596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:38.067986 596 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:38875
I20260620 08:14:38.068022 673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:38875 every 8 connection(s)
I20260620 08:14:38.068403 596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:38.069257 674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:38.071341 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 596
I20260620 08:14:38.071472 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:38.071676 25574 external_mini_cluster.cc:1428] Setting key 2a4bcfd4efdb482e787b2fcdce379a38
I20260620 08:14:38.071980 674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Bootstrap starting.
I20260620 08:14:38.072779 674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:38.073115 674 log.cc:824] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:38.073936 674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: No bootstrap required, opened a new log
I20260620 08:14:38.076381 674 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc25077815224057ac32edd836740f6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38875 } }
I20260620 08:14:38.076575 674 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:38.076627 674 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc25077815224057ac32edd836740f6c, State: Initialized, Role: FOLLOWER
I20260620 08:14:38.076752 674 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc25077815224057ac32edd836740f6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38875 } }
I20260620 08:14:38.076825 674 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:38.076855 674 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:38.076900 674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:38.077672 674 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc25077815224057ac32edd836740f6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38875 } }
I20260620 08:14:38.077785 674 leader_election.cc:302] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc25077815224057ac32edd836740f6c; no voters:
I20260620 08:14:38.077967 674 leader_election.cc:288] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:38.078032 679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:38.078224 679 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [term 1 LEADER]: Becoming Leader. State: Replica: cc25077815224057ac32edd836740f6c, State: Running, Role: LEADER
I20260620 08:14:38.078246 674 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:38.078337 679 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc25077815224057ac32edd836740f6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38875 } }
I20260620 08:14:38.078691 679 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc25077815224057ac32edd836740f6c. Latest consensus state: current_term: 1 leader_uuid: "cc25077815224057ac32edd836740f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc25077815224057ac32edd836740f6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38875 } } }
I20260620 08:14:38.078696 680 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc25077815224057ac32edd836740f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc25077815224057ac32edd836740f6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38875 } } }
I20260620 08:14:38.078779 679 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:38.078812 680 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c [sys.catalog]: This master's current role is: LEADER
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
W20260620 08:14:38.080531 695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:14:38.080605 695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:14:38.080660 687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:38.080834 687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:38.082489 687 catalog_manager.cc:1347] Generated new cluster ID: 543ae1284b0f4927b18f673e7ec0917f
I20260620 08:14:38.082541 687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:38.084244 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.072477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.24.249.190:38875)
0620 08:14:38.072700 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.072709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.072743 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:14:38.073366 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.073380 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.073430 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.073547 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.074314 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.075760 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.076562 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.077194 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.077316 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.080394 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.080398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.080398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.080411 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.082561 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.082984 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.082987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.082988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.083093 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.083323 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.083323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.083324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.083624 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.083794 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.083900 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.084035 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260620 08:14:38.087200 687 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:38.087680 687 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:38.090883 687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Generated new TSK 0
I20260620 08:14:38.091061 687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab.
Entry for principal oryx/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab.
I20260620 08:14:38.125108 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:38875
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab
--principal=oryx/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:38.223237 700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:38.223464 700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:38.223492 700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:38.224858 700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:38.224906 700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:38.224928 700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:38.224946 700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:38.226576 700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab
--principal=oryx/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:38.226869 700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:38.227105 700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:38.232335 706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:38.232336 708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:38.232647 705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:38.232736 700 server_base.cc:1034] running on GCE node
I20260620 08:14:38.233000 700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:38.233282 700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:38.234455 700 hybrid_clock.cc:648] HybridClock initialized: now 1781943278234405 us; error 82 us; skew 500 ppm
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:38.237121 700 init.cc:377] Logged in from keytab as oryx/127.24.249.129@KRBTEST.COM (short username oryx)
I20260620 08:14:38.237666 700 webserver.cc:460] Webserver started at http://127.24.249.129:38007/ using document root <none> and password file <none>
I20260620 08:14:38.237977 700 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:38.238054 700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:38.238238 700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:38.239415 700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "c7e7c6beecf44d23968db22ca2fd9c5c"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "87be77d38d4bbb79c0475a1e41b275c8"
server_key_iv: "94477b28f15526ce87690d2ebf610a52"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.239823 700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "c7e7c6beecf44d23968db22ca2fd9c5c"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "87be77d38d4bbb79c0475a1e41b275c8"
server_key_iv: "94477b28f15526ce87690d2ebf610a52"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.242003 700 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 08:14:38.242918 715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.243299 700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:38.243428 700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "c7e7c6beecf44d23968db22ca2fd9c5c"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "87be77d38d4bbb79c0475a1e41b275c8"
server_key_iv: "94477b28f15526ce87690d2ebf610a52"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.243520 700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:38.253970 700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:38.256345 700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:38.256541 700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:38.257205 700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:38.257258 700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.257292 700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:38.257311 700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:38.264796 700 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:33077
I20260620 08:14:38.264873 830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:33077 every 8 connection(s)
I20260620 08:14:38.265189 700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:38.267814 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.257656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:45543 (local address 127.24.249.190:38875)
0620 08:14:38.257728 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.257731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.257912 (+ 181us) server_negotiation.cc:407] Connection header received
0620 08:14:38.258568 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.258569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.258583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.258660 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.259390 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.260148 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.261396 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.261640 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.261726 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.264635 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.264637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.264637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.264645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.266438 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.266852 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.266854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.266854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.266911 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.267204 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.267205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.267205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.267324 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.267386 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.267664 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.267740 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:14:38.267964 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.257627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:38875 (local address 127.24.249.129:45543)
0620 08:14:38.257770 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.257810 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.258455 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.258808 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.258813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.259098 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:38.259290 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.259294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.260286 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.260287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.261288 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.261291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.261738 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.261739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.261885 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.262507 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:38.262516 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:38.264476 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:38.266528 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.266534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.266536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.266763 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.267003 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.267004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.267004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.267110 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.267406 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:38.267412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:38.267540 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.267634 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.267765 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260620 08:14:38.268484 831 heartbeater.cc:347] Connected to a master server at 127.24.249.190:38875
I20260620 08:14:38.268585 831 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:38.268774 831 heartbeater.cc:511] Master 127.24.249.190:38875 requested a full tablet report, sending...
I20260620 08:14:38.269388 627 ts_manager.cc:194] Registered new tserver with Master: c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077)
I20260620 08:14:38.270416 627 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:45543
I20260620 08:14:38.271067 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 700
I20260620 08:14:38.271183 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:38.271410 25574 external_mini_cluster.cc:1428] Setting key ad945df9a7619153ea6d70346b985fe2
I20260620 08:14:38.276556 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.269756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:37483 (local address 127.24.249.190:38875)
0620 08:14:38.269797 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.269799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.269942 (+ 143us) server_negotiation.cc:407] Connection header received
0620 08:14:38.270008 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.270009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.270017 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.270057 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.270417 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.271160 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.272154 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.272417 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.272481 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.273705 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.273706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.273706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.273712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.275330 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.275711 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.275713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.275713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.275802 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.276034 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.276035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.276035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.276156 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.276236 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.276381 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.276460 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:14:38.276561 837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.269692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:38875 (local address 127.24.249.129:37483)
0620 08:14:38.269883 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.269895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.269955 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.270069 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.270070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.270270 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:38.270330 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.270331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.271286 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.271287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.272020 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.272022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.272501 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.272502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.272575 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.273149 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:38.273155 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:38.273575 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:38.275416 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.275417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.275417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.275603 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.275859 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.275860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.275860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.275951 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.276228 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:38.276228 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:38.276279 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.276352 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.276460 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":36,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab.
Entry for principal oryx/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab.
I20260620 08:14:38.300406 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:38875
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab
--principal=oryx/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:38.384467 840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:38.384682 840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:38.384707 840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:38.386009 840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:38.386051 840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:38.386072 840 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:38.386089 840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:38.387624 840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab
--principal=oryx/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:38.387873 840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:38.388105 840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:38.393277 848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:38.393281 846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:38.393252 845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:38.393486 840 server_base.cc:1034] running on GCE node
I20260620 08:14:38.393687 840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:38.393950 840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:38.395103 840 hybrid_clock.cc:648] HybridClock initialized: now 1781943278395070 us; error 48 us; skew 500 ppm
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:38.397809 840 init.cc:377] Logged in from keytab as oryx/127.24.249.130@KRBTEST.COM (short username oryx)
I20260620 08:14:38.398288 840 webserver.cc:460] Webserver started at http://127.24.249.130:40589/ using document root <none> and password file <none>
I20260620 08:14:38.398530 840 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:38.398598 840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:38.398753 840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:38.400079 840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "d3cad2654db24bee9c365a7e8a36dd14"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "290265ff8d35e22df2605f204ed2a10e"
server_key_iv: "4c5c0765a42a1862749369dbfde7e0d1"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.400547 840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "d3cad2654db24bee9c365a7e8a36dd14"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "290265ff8d35e22df2605f204ed2a10e"
server_key_iv: "4c5c0765a42a1862749369dbfde7e0d1"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.405093 840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 08:14:38.408275 855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.409224 840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260620 08:14:38.409345 840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "d3cad2654db24bee9c365a7e8a36dd14"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "290265ff8d35e22df2605f204ed2a10e"
server_key_iv: "4c5c0765a42a1862749369dbfde7e0d1"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.409453 840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:38.431389 840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:38.434031 840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:38.434162 840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:38.434684 840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:38.434723 840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.434753 840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:38.434798 840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:38.442507 840 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:46271
I20260620 08:14:38.442588 970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:46271 every 8 connection(s)
I20260620 08:14:38.442996 840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:38.445619 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.435261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:51037 (local address 127.24.249.190:38875)
0620 08:14:38.435345 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.435349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.435513 (+ 164us) server_negotiation.cc:407] Connection header received
0620 08:14:38.436029 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.436030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.436043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.436145 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.437279 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.438183 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.439357 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.439561 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.439672 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.442389 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.442391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.442391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.442400 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.443991 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.444450 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.444451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.444451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.444510 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.444938 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.444939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.444940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.445069 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.445144 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.445476 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.445552 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:14:38.445803 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.435183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:38875 (local address 127.24.249.130:51037)
0620 08:14:38.435374 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.435415 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.435921 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.436268 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.436272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.436728 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:38.437139 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.437147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.438286 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.438287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.439250 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.439253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.439699 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.439700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.439840 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.440445 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:38.440458 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:38.442284 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:38.444112 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.444118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.444120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.444350 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.444630 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.444631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.444632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.444782 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.445168 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:38.445175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:38.445314 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.445417 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.445559 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260620 08:14:38.446424 971 heartbeater.cc:347] Connected to a master server at 127.24.249.190:38875
I20260620 08:14:38.446504 971 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:38.446702 971 heartbeater.cc:511] Master 127.24.249.190:38875 requested a full tablet report, sending...
I20260620 08:14:38.447150 628 ts_manager.cc:194] Registered new tserver with Master: d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271)
I20260620 08:14:38.447624 628 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:51037
I20260620 08:14:38.448839 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 840
I20260620 08:14:38.448966 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:38.449177 25574 external_mini_cluster.cc:1428] Setting key 03284fd5a71fc807d84a750a64f88b24
I20260620 08:14:38.454465 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.448357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:38875 (local address 127.24.249.130:43993)
0620 08:14:38.448523 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.448534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.448649 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.448802 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.448803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.449044 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:38.449117 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.449119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.449809 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.449810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.450399 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.450401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.450776 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.450777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.450843 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.451329 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:38.451334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:38.451636 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:38.453290 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.453291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.453292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.453534 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.453740 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.453741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.453741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.453814 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.454172 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:38.454173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:38.454246 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.454318 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.454396 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260620 08:14:38.454872 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.447984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:43993 (local address 127.24.249.190:38875)
0620 08:14:38.448050 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.448052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.448651 (+ 599us) server_negotiation.cc:407] Connection header received
0620 08:14:38.448711 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.448711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.448721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.448797 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.449198 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.449713 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.450495 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.450690 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.450774 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.451745 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.451746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.451746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.451753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.453211 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.453615 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.453616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.453616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.453668 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.453913 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.453914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.453914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.454087 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.454177 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.454366 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.454459 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for oryx/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab.
Entry for principal oryx/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab.
I20260620 08:14:38.478437 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:38875
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab
--principal=oryx/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:38.561568 980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:38.561784 980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:38.561808 980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:38.563012 980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:38.563059 980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:38.563079 980 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 08:14:38.563097 980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:38.564671 980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab
--principal=oryx/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.249.190:38875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:38.564934 980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:38.565182 980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:38.570628 988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:38.570629 986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:38.570629 985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:38.570853 980 server_base.cc:1034] running on GCE node
I20260620 08:14:38.571066 980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:38.571372 980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:38.572517 980 hybrid_clock.cc:648] HybridClock initialized: now 1781943278572490 us; error 48 us; skew 500 ppm
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:38.575215 980 init.cc:377] Logged in from keytab as oryx/127.24.249.131@KRBTEST.COM (short username oryx)
I20260620 08:14:38.575695 980 webserver.cc:460] Webserver started at http://127.24.249.131:36199/ using document root <none> and password file <none>
I20260620 08:14:38.575918 980 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:38.575963 980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:38.576116 980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:38.577562 980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "50bbd457c8624e009bfd30e93ee5552a"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "c1f589fed24510056d5f779bd43c2945"
server_key_iv: "cbd4f94b4c0e8120775717f3f5075542"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.577988 980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "50bbd457c8624e009bfd30e93ee5552a"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "c1f589fed24510056d5f779bd43c2945"
server_key_iv: "cbd4f94b4c0e8120775717f3f5075542"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.579608 980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:38.580427 995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.580650 980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:38.580750 980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "50bbd457c8624e009bfd30e93ee5552a"
format_stamp: "Formatted at 2026-06-20 08:14:38 on dist-test-slave-65w8"
server_key: "c1f589fed24510056d5f779bd43c2945"
server_key_iv: "cbd4f94b4c0e8120775717f3f5075542"
server_key_version: "encryptionkey@0"
I20260620 08:14:38.580823 980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:38.592811 980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:38.595100 980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:38.595271 980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:38.595791 980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:38.595826 980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.595854 980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:38.595906 980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:38.603121 980 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:41309
I20260620 08:14:38.603184 1110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:41309 every 8 connection(s)
I20260620 08:14:38.603564 980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:38.603859 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 980
I20260620 08:14:38.603961 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:38.604179 25574 external_mini_cluster.cc:1428] Setting key ebdfa3d4f86f3a2f47755db1fe16036f
I20260620 08:14:38.605679 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.596324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:49385 (local address 127.24.249.190:38875)
0620 08:14:38.596386 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.596388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.596583 (+ 195us) server_negotiation.cc:407] Connection header received
0620 08:14:38.597177 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.597179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.597191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.597249 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.598104 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.598663 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.599684 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.599888 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.599972 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.602592 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.602594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.602594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.602603 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.604268 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.604746 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.604747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.604747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.604806 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.605058 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.605059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.605059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.605178 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.605257 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.605512 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.605590 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:14:38.605859 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.596314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:38875 (local address 127.24.249.131:49385)
0620 08:14:38.596466 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.596501 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.597051 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.597320 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.597325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.597709 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:38.598009 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.598014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.598756 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.598757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.599571 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.599573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.599980 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.599981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.600116 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.600618 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:38.600642 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:38.602487 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:38.604367 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.604373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.604375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.604654 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.604880 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.604881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.604881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.604991 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.605247 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:38.605252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:38.605385 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.605492 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.605640 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260620 08:14:38.606319 1111 heartbeater.cc:347] Connected to a master server at 127.24.249.190:38875
I20260620 08:14:38.606403 1111 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:38.606571 1111 heartbeater.cc:511] Master 127.24.249.190:38875 requested a full tablet report, sending...
I20260620 08:14:38.606925 628 ts_manager.cc:194] Registered new tserver with Master: 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309)
I20260620 08:14:38.607321 628 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:49385
I20260620 08:14:38.612581 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.607349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:47389 (local address 127.24.249.190:38875)
0620 08:14:38.607389 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.607390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.607473 (+ 83us) server_negotiation.cc:407] Connection header received
0620 08:14:38.607542 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.607543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.607553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.607606 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.607921 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.608374 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.609076 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.609278 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.609349 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.610219 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.610221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.610221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.610228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.611445 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.611822 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.611824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.611824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.611878 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.612078 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.612079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.612079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.612190 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.612265 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.612417 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.612488 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:38.612593 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.607291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:38875 (local address 127.24.249.131:47389)
0620 08:14:38.607397 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.607407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.607485 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.607611 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.607612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.607791 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:38.607839 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.607840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.608463 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.608464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.608981 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.608982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.609360 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.609360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.609427 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.609784 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:38.609788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:38.610096 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:38.611569 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.611570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.611570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.611740 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.611952 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:38.611953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:38.611954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:38.612014 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:38.612258 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:38.612259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:38.612305 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.612391 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.612490 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260620 08:14:38.615835 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:14:38.637423 677 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:38.631763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.24.249.190:38875)
0620 08:14:38.631835 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.631837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.631848 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:38.631883 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.631884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.631893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.631946 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.632521 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.633044 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.633913 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.634102 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.634156 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.637299 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51096: BlockingRecv error: recv got EOF from 127.0.0.1:51096 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 08:14:38 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:38.644940 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.638181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.24.249.190:38875)
0620 08:14:38.638229 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.638231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.638322 (+ 91us) server_negotiation.cc:407] Connection header received
0620 08:14:38.638359 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.638360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.638368 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.638415 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.638845 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.639305 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.639934 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.640104 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.640175 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.642387 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:38.642388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:38.642389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:38.642397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:38.643770 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.644142 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.644144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.644144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.644211 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:38.644404 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:38.644406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:38.644406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:38.644520 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:38.644634 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.644756 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.644835 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:14:38.646847 628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:38.648581 628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:38.655391 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.651096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:33077 (local address 127.0.0.1:36252)
0620 08:14:38.651397 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.651440 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.651503 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.651876 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.651877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.651879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:38.651936 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.651938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.653240 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.653241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.654379 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.654381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.655075 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.655076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.655166 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.655180 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.655252 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.655303 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":5,"thread_start_us":47,"threads_started":1}
I20260620 08:14:38.655416 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.651222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.24.249.129:33077)
0620 08:14:38.651504 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.651508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.651520 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:38.651632 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.651635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.651653 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.651729 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:38.652054 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.653138 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.654473 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.654971 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.655091 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.655151 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.655266 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.655305 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":74,"threads_started":1}
I20260620 08:14:38.655591 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.651375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.24.249.131:41309)
0620 08:14:38.651552 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.651556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.651640 (+ 84us) server_negotiation.cc:407] Connection header received
0620 08:14:38.651730 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.651732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.651747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.651817 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:38.652171 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.652838 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.654414 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.655086 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.655172 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.655219 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.655454 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.655491 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260620 08:14:38.655714 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.651232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.24.249.130:46271)
0620 08:14:38.651423 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.651428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.651590 (+ 162us) server_negotiation.cc:407] Connection header received
0620 08:14:38.651636 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.651639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.651665 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.651735 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:38.651976 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.653169 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.654307 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.655034 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.655127 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.655185 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.655559 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.655598 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260620 08:14:38.655876 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.651291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:41309 (local address 127.0.0.1:54894)
0620 08:14:38.651571 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.651581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.651655 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.651830 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.651830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.651832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:38.652109 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.652111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.652924 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.652926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.654332 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.654334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.655247 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.655247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.655307 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.655312 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.655365 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.655413 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":58,"thread_start_us":79,"threads_started":1}
I20260620 08:14:38.655493 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.651132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46271 (local address 127.0.0.1:35230)
0620 08:14:38.651408 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.651439 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.651503 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.651813 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.651816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.651818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:38.651909 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.651912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.653256 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.653257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.654227 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.654230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.655222 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.655223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.655308 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.655313 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.655365 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.655412 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260620 08:14:38.656530 1045 tablet_service.cc:1467] Processing CreateTablet for tablet a2dd36eea122430fb675651dc14ff7e4 (DEFAULT_TABLE table=test-table [id=0315b707ec5248a79179f025cfaa667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:38.656560 765 tablet_service.cc:1467] Processing CreateTablet for tablet a2dd36eea122430fb675651dc14ff7e4 (DEFAULT_TABLE table=test-table [id=0315b707ec5248a79179f025cfaa667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:38.656826 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2dd36eea122430fb675651dc14ff7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:38.656908 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2dd36eea122430fb675651dc14ff7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:38.658564 905 tablet_service.cc:1467] Processing CreateTablet for tablet a2dd36eea122430fb675651dc14ff7e4 (DEFAULT_TABLE table=test-table [id=0315b707ec5248a79179f025cfaa667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:38.658882 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2dd36eea122430fb675651dc14ff7e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:38.660007 1139 tablet_bootstrap.cc:492] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Bootstrap starting.
I20260620 08:14:38.660531 1140 tablet_bootstrap.cc:492] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Bootstrap starting.
I20260620 08:14:38.660710 1139 tablet_bootstrap.cc:654] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:38.661056 1139 log.cc:824] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:38.661161 1140 tablet_bootstrap.cc:654] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:38.661477 1140 log.cc:824] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:38.661918 1139 tablet_bootstrap.cc:492] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: No bootstrap required, opened a new log
I20260620 08:14:38.661980 1139 ts_tablet_manager.cc:1397] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:14:38.662228 1140 tablet_bootstrap.cc:492] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: No bootstrap required, opened a new log
I20260620 08:14:38.662309 1140 ts_tablet_manager.cc:1397] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:38.662634 1143 tablet_bootstrap.cc:492] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Bootstrap starting.
I20260620 08:14:38.663295 1143 tablet_bootstrap.cc:654] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:38.663641 1143 log.cc:824] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:38.664048 1139 raft_consensus.cc:348] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.664232 1139 raft_consensus.cc:374] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:38.664345 1139 raft_consensus.cc:729] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Initialized, Role: FOLLOWER
I20260620 08:14:38.664402 1143 tablet_bootstrap.cc:492] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: No bootstrap required, opened a new log
I20260620 08:14:38.664477 1143 ts_tablet_manager.cc:1397] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:38.664484 1139 consensus_queue.cc:260] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.664592 1140 raft_consensus.cc:348] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.664739 1139 ts_tablet_manager.cc:1428] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:14:38.664753 1140 raft_consensus.cc:374] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:38.664789 1140 raft_consensus.cc:729] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bbd457c8624e009bfd30e93ee5552a, State: Initialized, Role: FOLLOWER
I20260620 08:14:38.664897 831 heartbeater.cc:503] Master 127.24.249.190:38875 was elected leader, sending a full tablet report...
I20260620 08:14:38.664903 1140 consensus_queue.cc:260] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.665148 1140 ts_tablet_manager.cc:1428] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:14:38.665190 1111 heartbeater.cc:503] Master 127.24.249.190:38875 was elected leader, sending a full tablet report...
I20260620 08:14:38.666535 1143 raft_consensus.cc:348] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.666684 1143 raft_consensus.cc:374] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:38.666707 1143 raft_consensus.cc:729] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cad2654db24bee9c365a7e8a36dd14, State: Initialized, Role: FOLLOWER
I20260620 08:14:38.666806 1143 consensus_queue.cc:260] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.667006 1143 ts_tablet_manager.cc:1428] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:14:38.667061 971 heartbeater.cc:503] Master 127.24.249.190:38875 was elected leader, sending a full tablet report...
W20260620 08:14:38.693696 972 tablet.cc:2369] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:14:38.765964 832 tablet.cc:2369] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:38.846007 1145 raft_consensus.cc:482] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:38.846136 1145 raft_consensus.cc:504] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.846521 1145 leader_election.cc:288] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309), d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271)
I20260620 08:14:38.849591 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.846771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:37885 (local address 127.24.249.130:46271)
0620 08:14:38.846828 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.846830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.847000 (+ 170us) server_negotiation.cc:407] Connection header received
0620 08:14:38.847067 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.847068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.847078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.847151 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:38.847314 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.847854 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.848865 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.849230 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.849323 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.849332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.849480 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.849514 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:14:38.849607 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.846855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:33939 (local address 127.24.249.131:41309)
0620 08:14:38.846931 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.846934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.846969 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:14:38.847076 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.847077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.847087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.847144 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:38.847336 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.847853 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.848863 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.849240 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.849340 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.849348 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.849494 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.849528 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:14:38.849622 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.846694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46271 (local address 127.24.249.129:37885)
0620 08:14:38.846904 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.846916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.847007 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.847154 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.847155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.847157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:38.847239 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.847240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.847947 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.847948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.848768 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.848770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.849322 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.849323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.849401 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.849405 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.849473 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.849514 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":16,"thread_start_us":101,"threads_started":1}
I20260620 08:14:38.849630 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.846774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:41309 (local address 127.24.249.129:33939)
0620 08:14:38.846904 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.846916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.846997 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.847149 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.847149 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.847151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:38.847230 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.847232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.847943 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.847944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.848762 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.848764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.849342 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.849342 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.849419 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.849422 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.849489 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.849528 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":39,"thread_start_us":70,"threads_started":1}
I20260620 08:14:38.850174 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2dd36eea122430fb675651dc14ff7e4" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14" is_pre_election: true
I20260620 08:14:38.850219 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2dd36eea122430fb675651dc14ff7e4" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a" is_pre_election: true
I20260620 08:14:38.850344 925 raft_consensus.cc:2454] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 0.
I20260620 08:14:38.850363 1065 raft_consensus.cc:2454] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 0.
I20260620 08:14:38.850564 719 leader_election.cc:302] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bbd457c8624e009bfd30e93ee5552a, c7e7c6beecf44d23968db22ca2fd9c5c; no voters:
I20260620 08:14:38.850698 1145 raft_consensus.cc:2781] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:38.850754 1145 raft_consensus.cc:482] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:38.850777 1145 raft_consensus.cc:3037] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:38.851460 1145 raft_consensus.cc:504] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.851598 1145 leader_election.cc:288] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 election: Requested vote from peers 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309), d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271)
I20260620 08:14:38.851764 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2dd36eea122430fb675651dc14ff7e4" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14"
I20260620 08:14:38.851742 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2dd36eea122430fb675651dc14ff7e4" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a"
I20260620 08:14:38.851840 925 raft_consensus.cc:3037] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:38.851866 1065 raft_consensus.cc:3037] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:38.852497 925 raft_consensus.cc:2454] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 1.
I20260620 08:14:38.852676 716 leader_election.cc:302] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7e7c6beecf44d23968db22ca2fd9c5c, d3cad2654db24bee9c365a7e8a36dd14; no voters:
I20260620 08:14:38.852718 1065 raft_consensus.cc:2454] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 1.
I20260620 08:14:38.852809 1145 raft_consensus.cc:2781] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:38.852953 1145 raft_consensus.cc:686] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 LEADER]: Becoming Leader. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Running, Role: LEADER
I20260620 08:14:38.853180 1145 consensus_queue.cc:237] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } }
I20260620 08:14:38.854074 627 catalog_manager.cc:5495] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c reported cstate change: term changed from 0 to 1, leader changed from <none> to c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129). New cstate: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } health_report { overall_health: UNKNOWN } } }
W20260620 08:14:38.854259 1112 tablet.cc:2369] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:14:38.886288 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.883833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.24.249.129:33077)
0620 08:14:38.883902 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.883904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.883916 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:38.883963 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.883963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.883987 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.884068 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:38.884250 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.884730 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.885487 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.885679 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.885759 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.885958 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:38.886057 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:38.886150 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.886168 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.886189 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:14:38.888350 925 raft_consensus.cc:1261] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Refusing update from remote peer c7e7c6beecf44d23968db22ca2fd9c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:38.888350 1065 raft_consensus.cc:1261] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Refusing update from remote peer c7e7c6beecf44d23968db22ca2fd9c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:38.888710 1150 consensus_queue.cc:1035] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:38.888890 1145 consensus_queue.cc:1035] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:38.891901 1154 mvcc.cc:205] Tried to move back new op lower bound from 7298839670323036160 to 7298839670183309312. Current Snapshot: MvccSnapshot[applied={T|T < 7298839670323036160}]
I20260620 08:14:38.892264 1155 mvcc.cc:205] Tried to move back new op lower bound from 7298839670323036160 to 7298839670183309312. Current Snapshot: MvccSnapshot[applied={T|T < 7298839670323036160}]
I20260620 08:14:38.894107 627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:38.894229 627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:38.895097 627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Sending DeleteTablet for 3 replicas of tablet a2dd36eea122430fb675651dc14ff7e4
I20260620 08:14:38.895450 765 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dd36eea122430fb675651dc14ff7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:38 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:36252
I20260620 08:14:38.895449 905 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dd36eea122430fb675651dc14ff7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:38 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:35230
I20260620 08:14:38.895483 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet a2dd36eea122430fb675651dc14ff7e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:38 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:54894
I20260620 08:14:38.895910 1164 tablet_replica.cc:330] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: stopping tablet replica
I20260620 08:14:38.895874 627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 08:14:38.896019 1164 raft_consensus.cc:2229] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 08:14:38.896035 627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:38.896128 1164 raft_consensus.cc:2258] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:38.896582 1164 ts_tablet_manager.cc:1905] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:38.896921 1162 tablet_replica.cc:330] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: stopping tablet replica
I20260620 08:14:38.897069 1162 raft_consensus.cc:2229] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:38.897281 1163 tablet_replica.cc:330] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: stopping tablet replica
I20260620 08:14:38.897274 905 tablet_service.cc:1467] Processing CreateTablet for tablet 29c64990f0e44d8bbc99378e193d6cb7 (DEFAULT_TABLE table=test-table [id=d65142528c4a441f80961cf91963a089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:38.897380 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c64990f0e44d8bbc99378e193d6cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:38.897430 765 tablet_service.cc:1467] Processing CreateTablet for tablet 29c64990f0e44d8bbc99378e193d6cb7 (DEFAULT_TABLE table=test-table [id=d65142528c4a441f80961cf91963a089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:38.897547 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c64990f0e44d8bbc99378e193d6cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:38.897696 1163 raft_consensus.cc:2229] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:38.897770 1162 raft_consensus.cc:2258] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:38.897996 1163 raft_consensus.cc:2258] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:38.898303 1164 ts_tablet_manager.cc:1918] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:38.898378 1164 log.cc:1196] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/a2dd36eea122430fb675651dc14ff7e4
I20260620 08:14:38.898635 1164 ts_tablet_manager.cc:1939] T a2dd36eea122430fb675651dc14ff7e4 P d3cad2654db24bee9c365a7e8a36dd14: Deleting consensus metadata
I20260620 08:14:38.898957 1143 tablet_bootstrap.cc:492] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: Bootstrap starting.
I20260620 08:14:38.899231 1139 tablet_bootstrap.cc:492] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: Bootstrap starting.
I20260620 08:14:38.899251 612 catalog_manager.cc:4841] TS d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271): tablet a2dd36eea122430fb675651dc14ff7e4 (table test-table [id=0315b707ec5248a79179f025cfaa667f]) successfully deleted
I20260620 08:14:38.899362 1143 tablet_bootstrap.cc:654] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:38.897331 1045 tablet_service.cc:1467] Processing CreateTablet for tablet 29c64990f0e44d8bbc99378e193d6cb7 (DEFAULT_TABLE table=test-table [id=d65142528c4a441f80961cf91963a089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:38.899500 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c64990f0e44d8bbc99378e193d6cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:38.899688 1163 ts_tablet_manager.cc:1905] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:38.899725 1139 tablet_bootstrap.cc:654] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:38.900175 1162 ts_tablet_manager.cc:1905] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:38.900981 1140 tablet_bootstrap.cc:492] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: Bootstrap starting.
I20260620 08:14:38.901300 1163 ts_tablet_manager.cc:1918] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:38.901363 1163 log.cc:1196] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/a2dd36eea122430fb675651dc14ff7e4
I20260620 08:14:38.901392 1140 tablet_bootstrap.cc:654] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:38.901619 1163 ts_tablet_manager.cc:1939] T a2dd36eea122430fb675651dc14ff7e4 P 50bbd457c8624e009bfd30e93ee5552a: Deleting consensus metadata
I20260620 08:14:38.901799 1162 ts_tablet_manager.cc:1918] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:38.901898 1162 log.cc:1196] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/a2dd36eea122430fb675651dc14ff7e4
I20260620 08:14:38.902050 615 catalog_manager.cc:4841] TS 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309): tablet a2dd36eea122430fb675651dc14ff7e4 (table test-table [id=0315b707ec5248a79179f025cfaa667f]) successfully deleted
I20260620 08:14:38.902164 1162 ts_tablet_manager.cc:1939] T a2dd36eea122430fb675651dc14ff7e4 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting consensus metadata
I20260620 08:14:38.902174 1140 tablet_bootstrap.cc:492] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: No bootstrap required, opened a new log
I20260620 08:14:38.902227 1140 ts_tablet_manager.cc:1397] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:38.902380 1140 raft_consensus.cc:348] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.902477 1140 raft_consensus.cc:374] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:38.902500 1140 raft_consensus.cc:729] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bbd457c8624e009bfd30e93ee5552a, State: Initialized, Role: FOLLOWER
I20260620 08:14:38.902541 1140 consensus_queue.cc:260] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.902631 1140 ts_tablet_manager.cc:1428] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:38.902712 615 catalog_manager.cc:4841] TS c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077): tablet a2dd36eea122430fb675651dc14ff7e4 (table test-table [id=0315b707ec5248a79179f025cfaa667f]) successfully deleted
I20260620 08:14:38.909927 1143 tablet_bootstrap.cc:492] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: No bootstrap required, opened a new log
I20260620 08:14:38.909965 1139 tablet_bootstrap.cc:492] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: No bootstrap required, opened a new log
I20260620 08:14:38.910004 1143 ts_tablet_manager.cc:1397] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 08:14:38.910025 1139 ts_tablet_manager.cc:1397] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 08:14:38.910187 1139 raft_consensus.cc:348] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.910185 1143 raft_consensus.cc:348] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.910279 1139 raft_consensus.cc:374] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:38.910280 1143 raft_consensus.cc:374] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:38.910301 1139 raft_consensus.cc:729] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Initialized, Role: FOLLOWER
I20260620 08:14:38.910306 1143 raft_consensus.cc:729] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cad2654db24bee9c365a7e8a36dd14, State: Initialized, Role: FOLLOWER
I20260620 08:14:38.910353 1139 consensus_queue.cc:260] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.910353 1143 consensus_queue.cc:260] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.910457 1139 ts_tablet_manager.cc:1428] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.910457 1143 ts_tablet_manager.cc:1428] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:38.955080 1146 raft_consensus.cc:482] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:38.955207 1146 raft_consensus.cc:504] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.955540 1146 leader_election.cc:288] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077), d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271)
I20260620 08:14:38.958830 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.955763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46271 (local address 127.24.249.131:59423)
0620 08:14:38.955827 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.955841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.955887 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.956085 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.956087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.956089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:38.956158 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.956160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.957041 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.957042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.957925 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.957927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.958524 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.958525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.958624 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.958629 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.958695 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.958737 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 08:14:38.958830 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.955848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:59423 (local address 127.24.249.130:46271)
0620 08:14:38.955889 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.955891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.955936 (+ 45us) server_negotiation.cc:407] Connection header received
0620 08:14:38.955964 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.955965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.955974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.956033 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:38.956236 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.956952 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.958020 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.958435 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.958507 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.958516 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.958717 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.958756 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:14:38.959111 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c64990f0e44d8bbc99378e193d6cb7" candidate_uuid: "50bbd457c8624e009bfd30e93ee5552a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14" is_pre_election: true
I20260620 08:14:38.959214 925 raft_consensus.cc:2454] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bbd457c8624e009bfd30e93ee5552a in term 0.
I20260620 08:14:38.959247 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.955924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:60759 (local address 127.24.249.129:33077)
0620 08:14:38.955965 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.955967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.956010 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:14:38.956131 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.956132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.956167 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.956221 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:38.956442 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.957056 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.958370 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.958850 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.958921 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.958929 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:38.959133 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 08:14:38.959168 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:14:38.959393 996 leader_election.cc:302] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bbd457c8624e009bfd30e93ee5552a, d3cad2654db24bee9c365a7e8a36dd14; no voters:
I20260620 08:14:38.959519 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.955774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:33077 (local address 127.24.249.131:60759)
0620 08:14:38.955959 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.955968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.956034 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.956266 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.956267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.956269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:38.956361 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.956363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.957150 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.957151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.958266 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.958268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:38.958965 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:38.958966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:38.959055 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:38.959060 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:38.959325 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 08:14:38.959378 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260620 08:14:38.959542 1146 raft_consensus.cc:2781] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:38.959574 1146 raft_consensus.cc:482] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:38.959595 1146 raft_consensus.cc:3037] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:38.959798 785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c64990f0e44d8bbc99378e193d6cb7" candidate_uuid: "50bbd457c8624e009bfd30e93ee5552a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" is_pre_election: true
I20260620 08:14:38.959942 785 raft_consensus.cc:2454] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50bbd457c8624e009bfd30e93ee5552a in term 0.
I20260620 08:14:38.960502 1146 raft_consensus.cc:504] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.960662 1146 leader_election.cc:288] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [CANDIDATE]: Term 1 election: Requested vote from peers c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077), d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271)
I20260620 08:14:38.960831 785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c64990f0e44d8bbc99378e193d6cb7" candidate_uuid: "50bbd457c8624e009bfd30e93ee5552a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c"
I20260620 08:14:38.960901 785 raft_consensus.cc:3037] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:38.960932 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c64990f0e44d8bbc99378e193d6cb7" candidate_uuid: "50bbd457c8624e009bfd30e93ee5552a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14"
I20260620 08:14:38.961010 925 raft_consensus.cc:3037] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:38.961526 785 raft_consensus.cc:2454] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bbd457c8624e009bfd30e93ee5552a in term 1.
I20260620 08:14:38.961694 999 leader_election.cc:302] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bbd457c8624e009bfd30e93ee5552a, c7e7c6beecf44d23968db22ca2fd9c5c; no voters:
I20260620 08:14:38.961781 925 raft_consensus.cc:2454] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50bbd457c8624e009bfd30e93ee5552a in term 1.
I20260620 08:14:38.961973 1146 raft_consensus.cc:2781] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:38.962148 1146 raft_consensus.cc:686] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 1 LEADER]: Becoming Leader. State: Replica: 50bbd457c8624e009bfd30e93ee5552a, State: Running, Role: LEADER
I20260620 08:14:38.962234 1146 consensus_queue.cc:237] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:38.963001 627 catalog_manager.cc:5495] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a reported cstate change: term changed from 0 to 1, leader changed from <none> to 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131). New cstate: current_term: 1 leader_uuid: "50bbd457c8624e009bfd30e93ee5552a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } health_report { overall_health: HEALTHY } } }
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943278, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.190@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:39.006898 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.998866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.24.249.190:38875)
0620 08:14:38.998924 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:38.998927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:38.999077 (+ 150us) server_negotiation.cc:407] Connection header received
0620 08:14:38.999173 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:38.999174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:38.999186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:38.999268 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:38.999755 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.000561 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.001359 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.001635 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.001718 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.004070 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:39.004071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:39.004071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:39.004077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:39.005558 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:39.006077 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:39.006078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:39.006078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:39.006129 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:39.006360 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:39.006360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:39.006360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:39.006461 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:39.006510 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.006678 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.006819 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:39.007115 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:38.998788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:38875 (local address 127.0.0.1:51116)
0620 08:14:38.998953 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:38.998967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:38.999097 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:38.999286 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:38.999287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:38.999564 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:38.999654 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:38.999656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.000673 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.000674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.001254 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.001256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.001762 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.001763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.001811 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.002485 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:39.002490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:39.003973 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:39.005658 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:39.005666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:39.005668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:39.005993 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:39.006224 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:39.006225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:39.006225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:39.006298 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:39.006521 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:39.006527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:39.006583 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 08:14:39.006663 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:14:39.006890 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":98,"thread_start_us":106,"threads_started":1}
I20260620 08:14:39.009373 627 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:51116:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 08:14:39.009634 627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:39.011117 905 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0a988483d947e0ada861b90e15cd19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03893a5d805e4d3e8319af4864ae4880]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:39.011132 765 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0a988483d947e0ada861b90e15cd19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03893a5d805e4d3e8319af4864ae4880]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:39.011129 1045 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0a988483d947e0ada861b90e15cd19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03893a5d805e4d3e8319af4864ae4880]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 08:14:39.011286 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0a988483d947e0ada861b90e15cd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.011286 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0a988483d947e0ada861b90e15cd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.011301 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0a988483d947e0ada861b90e15cd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.013011 1140 tablet_bootstrap.cc:492] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: Bootstrap starting.
I20260620 08:14:39.013006 1143 tablet_bootstrap.cc:492] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: Bootstrap starting.
I20260620 08:14:39.013468 1140 tablet_bootstrap.cc:654] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.013468 1143 tablet_bootstrap.cc:654] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.013895 1139 tablet_bootstrap.cc:492] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: Bootstrap starting.
I20260620 08:14:39.014171 1139 tablet_bootstrap.cc:654] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.014770 1139 tablet_bootstrap.cc:492] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: No bootstrap required, opened a new log
I20260620 08:14:39.014828 1139 ts_tablet_manager.cc:1397] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:39.014990 1143 tablet_bootstrap.cc:492] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: No bootstrap required, opened a new log
I20260620 08:14:39.014971 1139 raft_consensus.cc:348] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.015058 1139 raft_consensus.cc:374] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.015057 1143 ts_tablet_manager.cc:1397] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:39.015094 1139 raft_consensus.cc:729] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.015223 1143 raft_consensus.cc:348] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.015296 1143 raft_consensus.cc:374] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.015321 1143 raft_consensus.cc:729] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cad2654db24bee9c365a7e8a36dd14, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.015148 1139 consensus_queue.cc:260] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.015372 1143 consensus_queue.cc:260] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.015446 1139 ts_tablet_manager.cc:1428] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:39.015477 1143 ts_tablet_manager.cc:1428] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:39.015496 1140 tablet_bootstrap.cc:492] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: No bootstrap required, opened a new log
I20260620 08:14:39.015545 1140 ts_tablet_manager.cc:1397] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 08:14:39.015604 1145 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.015708 1145 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:39.015825 1140 raft_consensus.cc:348] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.015930 1140 raft_consensus.cc:374] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.015867 1147 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.015951 1140 raft_consensus.cc:729] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bbd457c8624e009bfd30e93ee5552a, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.015967 1147 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:39.015993 1140 consensus_queue.cc:260] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.016083 1140 ts_tablet_manager.cc:1428] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:39.016129 1146 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.016201 1146 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:39.110069 1145 raft_consensus.cc:482] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:39.110167 1145 raft_consensus.cc:504] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.110327 1145 leader_election.cc:288] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271), 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309)
I20260620 08:14:39.110520 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a988483d947e0ada861b90e15cd19" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14" is_pre_election: true
I20260620 08:14:39.110544 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a988483d947e0ada861b90e15cd19" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a" is_pre_election: true
I20260620 08:14:39.110620 925 raft_consensus.cc:2454] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 0.
I20260620 08:14:39.110621 1065 raft_consensus.cc:2454] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 0.
I20260620 08:14:39.110780 716 leader_election.cc:302] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7e7c6beecf44d23968db22ca2fd9c5c, d3cad2654db24bee9c365a7e8a36dd14; no voters:
I20260620 08:14:39.110910 1145 raft_consensus.cc:2781] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:39.110976 1145 raft_consensus.cc:482] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:39.110992 1145 raft_consensus.cc:3037] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.111712 1145 raft_consensus.cc:504] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.111892 1145 leader_election.cc:288] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 election: Requested vote from peers d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271), 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309)
I20260620 08:14:39.112041 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a988483d947e0ada861b90e15cd19" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a"
I20260620 08:14:39.112128 1065 raft_consensus.cc:3037] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.112142 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0a988483d947e0ada861b90e15cd19" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14"
I20260620 08:14:39.112233 925 raft_consensus.cc:3037] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.112920 1065 raft_consensus.cc:2454] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 1.
I20260620 08:14:39.112939 925 raft_consensus.cc:2454] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 1.
I20260620 08:14:39.113101 719 leader_election.cc:302] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bbd457c8624e009bfd30e93ee5552a, c7e7c6beecf44d23968db22ca2fd9c5c; no voters:
I20260620 08:14:39.113198 1145 raft_consensus.cc:2781] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:39.113289 1145 raft_consensus.cc:686] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 LEADER]: Becoming Leader. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Running, Role: LEADER
I20260620 08:14:39.113351 1145 consensus_queue.cc:237] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.113683 1150 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: TxnStatusTablet state changed. Reason: New leader c7e7c6beecf44d23968db22ca2fd9c5c. Latest consensus state: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.113780 1150 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:39.114004 1181 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 08:14:39.114126 1181 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 08:14:39.114069 628 catalog_manager.cc:5495] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c reported cstate change: term changed from 0 to 1, leader changed from <none> to c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129). New cstate: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:39.137519 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.135041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.24.249.129:33077)
0620 08:14:39.135106 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.135108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.135118 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:39.135226 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.135227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.135239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.135312 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:39.135450 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.135939 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.136746 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.136943 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.137028 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.137248 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.137290 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.137350 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.137417 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.137437 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 08:14:39.137522 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.134961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:33077 (local address 127.0.0.1:36256)
0620 08:14:39.135036 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:39.135051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:39.135123 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:39.135319 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:39.135320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:39.135321 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:39.135375 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.135382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.136033 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.136034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.136609 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.136611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.137030 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.137031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.137127 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.137144 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.137352 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.137361 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 08:14:39.137404 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 08:14:39.137452 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:14:39.138602 1065 raft_consensus.cc:1261] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Refusing update from remote peer c7e7c6beecf44d23968db22ca2fd9c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.138674 925 raft_consensus.cc:1261] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Refusing update from remote peer c7e7c6beecf44d23968db22ca2fd9c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.138967 1150 consensus_queue.cc:1035] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.139098 1145 consensus_queue.cc:1035] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.139509 1147 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: TxnStatusTablet state changed. Reason: New leader c7e7c6beecf44d23968db22ca2fd9c5c. Latest consensus state: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.139540 1146 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: TxnStatusTablet state changed. Reason: New leader c7e7c6beecf44d23968db22ca2fd9c5c. Latest consensus state: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.139631 1146 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:39.139632 1147 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:39.140399 1145 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.140467 1150 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.140547 1150 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:39.140596 1153 mvcc.cc:205] Tried to move back new op lower bound from 7298839671349837824 to 7298839671248625664. Current Snapshot: MvccSnapshot[applied={T|T < 7298839671349837824 or (T in {7298839671349837824})}]
I20260620 08:14:39.140882 1147 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.140967 1147 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P d3cad2654db24bee9c365a7e8a36dd14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:39.140479 1145 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P c7e7c6beecf44d23968db22ca2fd9c5c: This TxnStatusTablet replica's current role is: LEADER
I20260620 08:14:39.141573 1146 tablet_replica.cc:439] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } } }
I20260620 08:14:39.141672 1146 tablet_replica.cc:442] T 0f0a988483d947e0ada861b90e15cd19 P 50bbd457c8624e009bfd30e93ee5552a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 08:14:39.144444 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.142006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.24.249.131:41309)
0620 08:14:39.142050 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.142052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.142061 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:39.142094 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.142095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.142114 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.142193 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:39.142367 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.142854 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.143608 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.143839 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.143914 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.144092 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.144200 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.144284 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.144325 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.144346 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:14:39.150084 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.147084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:33077 (local address 127.24.249.131:51705)
0620 08:14:39.147258 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:39.147271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:39.147357 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:39.147483 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:39.147484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:39.147485 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:39.147562 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.147563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.148442 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.148443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.149087 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.149089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.149559 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.149560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.149607 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.149620 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.149824 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.149826 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:39.149966 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 08:14:39.150017 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260620 08:14:39.150074 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.147222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:51705 (local address 127.24.249.129:33077)
0620 08:14:39.147284 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.147286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.147319 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:14:39.147410 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.147410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.147424 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.147480 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:39.147620 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.148352 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.149170 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.149481 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.149551 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.149698 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.149730 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.149791 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.149971 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.149994 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:14:39.152282 925 raft_consensus.cc:1261] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Refusing update from remote peer 50bbd457c8624e009bfd30e93ee5552a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.152451 785 raft_consensus.cc:1261] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Refusing update from remote peer 50bbd457c8624e009bfd30e93ee5552a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.152730 1169 consensus_queue.cc:1035] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.152848 1146 consensus_queue.cc:1035] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.164721 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.158600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:41831 (local address 127.24.249.131:41309)
0620 08:14:39.158640 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.158642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.162079 (+ 3437us) server_negotiation.cc:407] Connection header received
0620 08:14:39.162133 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.162134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.162145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.162199 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:39.162342 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.162837 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.163997 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.164184 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.164248 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.164446 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.164476 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.164527 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.164645 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.164666 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:39.164750 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.158564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:41309 (local address 127.24.249.129:41831)
0620 08:14:39.161997 (+ 3433us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:39.162010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:39.162077 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:39.162203 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:39.162204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:39.162205 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:39.162275 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.162277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.162922 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.162923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.163853 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.163856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.164264 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.164265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.164323 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.164342 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.164544 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.164546 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:39.164602 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 08:14:39.164649 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3388,"thread_start_us":71,"threads_started":1}
I20260620 08:14:39.175138 627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:39.175259 627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:39.175887 627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Sending DeleteTablet for 3 replicas of tablet 29c64990f0e44d8bbc99378e193d6cb7
I20260620 08:14:39.176139 765 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c64990f0e44d8bbc99378e193d6cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:36252
I20260620 08:14:39.176195 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c64990f0e44d8bbc99378e193d6cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:54894
I20260620 08:14:39.176244 1162 tablet_replica.cc:330] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: stopping tablet replica
I20260620 08:14:39.176294 1163 tablet_replica.cc:330] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: stopping tablet replica
I20260620 08:14:39.176302 1162 raft_consensus.cc:2229] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:39.176337 1162 raft_consensus.cc:2258] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.176345 1163 raft_consensus.cc:2229] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:39.176420 1163 raft_consensus.cc:2258] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.176644 905 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c64990f0e44d8bbc99378e193d6cb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:35230
I20260620 08:14:39.176697 1162 ts_tablet_manager.cc:1905] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.176738 1164 tablet_replica.cc:330] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: stopping tablet replica
I20260620 08:14:39.176795 1164 raft_consensus.cc:2229] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:39.176834 1164 raft_consensus.cc:2258] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.177073 1163 ts_tablet_manager.cc:1905] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.177142 1164 ts_tablet_manager.cc:1905] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.178668 1162 ts_tablet_manager.cc:1918] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:39.178719 1162 log.cc:1196] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/29c64990f0e44d8bbc99378e193d6cb7
I20260620 08:14:39.178889 1164 ts_tablet_manager.cc:1918] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:39.178937 1164 log.cc:1196] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/29c64990f0e44d8bbc99378e193d6cb7
I20260620 08:14:39.178938 1162 ts_tablet_manager.cc:1939] T 29c64990f0e44d8bbc99378e193d6cb7 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting consensus metadata
I20260620 08:14:39.179157 1164 ts_tablet_manager.cc:1939] T 29c64990f0e44d8bbc99378e193d6cb7 P d3cad2654db24bee9c365a7e8a36dd14: Deleting consensus metadata
I20260620 08:14:39.179657 612 catalog_manager.cc:4841] TS d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271): tablet 29c64990f0e44d8bbc99378e193d6cb7 (table test-table [id=d65142528c4a441f80961cf91963a089]) successfully deleted
I20260620 08:14:39.180392 615 catalog_manager.cc:4841] TS c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077): tablet 29c64990f0e44d8bbc99378e193d6cb7 (table test-table [id=d65142528c4a441f80961cf91963a089]) successfully deleted
I20260620 08:14:39.182123 1163 ts_tablet_manager.cc:1918] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:39.182188 1163 log.cc:1196] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/29c64990f0e44d8bbc99378e193d6cb7
I20260620 08:14:39.182418 1163 ts_tablet_manager.cc:1939] T 29c64990f0e44d8bbc99378e193d6cb7 P 50bbd457c8624e009bfd30e93ee5552a: Deleting consensus metadata
I20260620 08:14:39.182936 615 catalog_manager.cc:4841] TS 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309): tablet 29c64990f0e44d8bbc99378e193d6cb7 (table test-table [id=d65142528c4a441f80961cf91963a089]) successfully deleted
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:14:39.209581 677 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:39.202445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.24.249.190:38875)
0620 08:14:39.202493 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.202495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.202667 (+ 172us) server_negotiation.cc:407] Connection header received
0620 08:14:39.202731 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.202732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.202754 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.202817 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:39.203567 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.204403 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.205539 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.205845 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.205963 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.209512 (+ 3549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51118: BlockingRecv error: recv got EOF from 127.0.0.1:51118 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM
I20260620 08:14:39.219789 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.210547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.24.249.190:38875)
0620 08:14:39.210585 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.210587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.210840 (+ 253us) server_negotiation.cc:407] Connection header received
0620 08:14:39.210893 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.210894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.210906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.210963 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:39.211543 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.212289 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.213373 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.213673 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.213760 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.216593 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:39.216595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:39.216595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:39.216602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:39.218402 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:39.218833 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:39.218834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:39.218835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:39.218896 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:39.219122 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:39.219123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:39.219123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:39.219264 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:39.219375 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.219527 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.219610 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:39.221531 627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:39.221715 627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:39.223112 765 tablet_service.cc:1467] Processing CreateTablet for tablet 687bc729a2f34db58b353ba7a0810b86 (DEFAULT_TABLE table=test-table [id=0d658aebcb11417186703616ccea8d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:39.223143 1045 tablet_service.cc:1467] Processing CreateTablet for tablet 687bc729a2f34db58b353ba7a0810b86 (DEFAULT_TABLE table=test-table [id=0d658aebcb11417186703616ccea8d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:39.223254 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687bc729a2f34db58b353ba7a0810b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.223268 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687bc729a2f34db58b353ba7a0810b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.224584 1139 tablet_bootstrap.cc:492] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: Bootstrap starting.
I20260620 08:14:39.224679 1140 tablet_bootstrap.cc:492] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: Bootstrap starting.
I20260620 08:14:39.224924 1139 tablet_bootstrap.cc:654] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.225064 1140 tablet_bootstrap.cc:654] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.225723 905 tablet_service.cc:1467] Processing CreateTablet for tablet 687bc729a2f34db58b353ba7a0810b86 (DEFAULT_TABLE table=test-table [id=0d658aebcb11417186703616ccea8d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:39.225843 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 687bc729a2f34db58b353ba7a0810b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.225979 1140 tablet_bootstrap.cc:492] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: No bootstrap required, opened a new log
I20260620 08:14:39.226020 1140 ts_tablet_manager.cc:1397] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:39.226171 1140 raft_consensus.cc:348] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.226235 1140 raft_consensus.cc:374] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.226254 1140 raft_consensus.cc:729] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bbd457c8624e009bfd30e93ee5552a, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.226297 1140 consensus_queue.cc:260] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.226377 1140 ts_tablet_manager.cc:1428] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:39.227061 1143 tablet_bootstrap.cc:492] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: Bootstrap starting.
I20260620 08:14:39.227504 1143 tablet_bootstrap.cc:654] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.228168 1143 tablet_bootstrap.cc:492] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: No bootstrap required, opened a new log
I20260620 08:14:39.228241 1143 ts_tablet_manager.cc:1397] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:39.228394 1143 raft_consensus.cc:348] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.228466 1143 raft_consensus.cc:374] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.228487 1143 raft_consensus.cc:729] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cad2654db24bee9c365a7e8a36dd14, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.228526 1143 consensus_queue.cc:260] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.228615 1143 ts_tablet_manager.cc:1428] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:39.230321 1139 tablet_bootstrap.cc:492] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: No bootstrap required, opened a new log
I20260620 08:14:39.230373 1139 ts_tablet_manager.cc:1397] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 08:14:39.230470 1139 raft_consensus.cc:348] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.230528 1139 raft_consensus.cc:374] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.230546 1139 raft_consensus.cc:729] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.230592 1139 consensus_queue.cc:260] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.230739 1139 ts_tablet_manager.cc:1428] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:39.335997 1147 raft_consensus.cc:482] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:39.336102 1147 raft_consensus.cc:504] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.336489 1147 leader_election.cc:288] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077), 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309)
I20260620 08:14:39.339871 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.336856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:34143 (local address 127.24.249.129:33077)
0620 08:14:39.336911 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.336913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.337012 (+ 99us) server_negotiation.cc:407] Connection header received
0620 08:14:39.337129 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.337132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.337163 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.337222 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:39.337424 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.337921 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.339132 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.339497 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.339583 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.339594 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.339769 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.339804 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:14:39.339886 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.336712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:33077 (local address 127.24.249.130:34143)
0620 08:14:39.336915 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:39.336927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:39.337027 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:39.337243 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:39.337245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:39.337248 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:39.337337 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.337339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.338013 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.338014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.338976 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.338979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.339601 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.339602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.339682 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.339685 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:39.339759 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:14:39.339800 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
I20260620 08:14:39.340098 785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687bc729a2f34db58b353ba7a0810b86" candidate_uuid: "d3cad2654db24bee9c365a7e8a36dd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" is_pre_election: true
I20260620 08:14:39.340149 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.336955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:32803 (local address 127.24.249.131:41309)
0620 08:14:39.337008 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.337010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.337089 (+ 79us) server_negotiation.cc:407] Connection header received
0620 08:14:39.337212 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.337214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.337224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.337297 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:14:39.337469 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.338027 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.339385 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.339744 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.339831 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.339838 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.340015 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.340058 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:14:39.340189 785 raft_consensus.cc:2454] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cad2654db24bee9c365a7e8a36dd14 in term 0.
I20260620 08:14:39.340353 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.336882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:41309 (local address 127.24.249.130:32803)
0620 08:14:39.336971 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:39.336980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:39.337103 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:39.337300 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:39.337301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:39.337303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:14:39.337397 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.337398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.338142 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.338143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.339287 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.339289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.339829 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.339830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.339905 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.339908 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:39.340224 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 08:14:39.340265 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":42,"threads_started":1}
I20260620 08:14:39.340425 859 leader_election.cc:302] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7e7c6beecf44d23968db22ca2fd9c5c, d3cad2654db24bee9c365a7e8a36dd14; no voters:
I20260620 08:14:39.340560 1147 raft_consensus.cc:2781] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:39.340607 1147 raft_consensus.cc:482] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:39.340632 1147 raft_consensus.cc:3037] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.340706 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687bc729a2f34db58b353ba7a0810b86" candidate_uuid: "d3cad2654db24bee9c365a7e8a36dd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a" is_pre_election: true
I20260620 08:14:39.340826 1065 raft_consensus.cc:2454] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cad2654db24bee9c365a7e8a36dd14 in term 0.
I20260620 08:14:39.341432 1147 raft_consensus.cc:504] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.341584 1147 leader_election.cc:288] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [CANDIDATE]: Term 1 election: Requested vote from peers c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077), 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309)
I20260620 08:14:39.341727 785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687bc729a2f34db58b353ba7a0810b86" candidate_uuid: "d3cad2654db24bee9c365a7e8a36dd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c"
I20260620 08:14:39.341737 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "687bc729a2f34db58b353ba7a0810b86" candidate_uuid: "d3cad2654db24bee9c365a7e8a36dd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a"
I20260620 08:14:39.341789 785 raft_consensus.cc:3037] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.341805 1065 raft_consensus.cc:3037] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.342428 1065 raft_consensus.cc:2454] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cad2654db24bee9c365a7e8a36dd14 in term 1.
I20260620 08:14:39.342430 785 raft_consensus.cc:2454] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cad2654db24bee9c365a7e8a36dd14 in term 1.
I20260620 08:14:39.342599 859 leader_election.cc:302] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7e7c6beecf44d23968db22ca2fd9c5c, d3cad2654db24bee9c365a7e8a36dd14; no voters:
I20260620 08:14:39.342708 1147 raft_consensus.cc:2781] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:39.342864 1147 raft_consensus.cc:686] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 LEADER]: Becoming Leader. State: Replica: d3cad2654db24bee9c365a7e8a36dd14, State: Running, Role: LEADER
I20260620 08:14:39.342939 1147 consensus_queue.cc:237] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.343640 627 catalog_manager.cc:5495] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130). New cstate: current_term: 1 leader_uuid: "d3cad2654db24bee9c365a7e8a36dd14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:39.375227 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.372598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.24.249.130:46271)
0620 08:14:39.372671 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.372673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.372683 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:39.372731 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.372732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.372754 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.372820 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:39.372980 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.373470 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.374413 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.374611 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.374672 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.374820 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.374924 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.374980 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.375046 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.375064 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:14:39.376269 1065 raft_consensus.cc:1261] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Refusing update from remote peer d3cad2654db24bee9c365a7e8a36dd14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.376304 785 raft_consensus.cc:1261] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Refusing update from remote peer d3cad2654db24bee9c365a7e8a36dd14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.376573 1147 consensus_queue.cc:1035] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.376753 1213 consensus_queue.cc:1035] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.380642 628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:39.380735 628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:39.381419 628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Sending DeleteTablet for 3 replicas of tablet 687bc729a2f34db58b353ba7a0810b86
I20260620 08:14:39.381637 765 tablet_service.cc:1514] Processing DeleteTablet for tablet 687bc729a2f34db58b353ba7a0810b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:36252
I20260620 08:14:39.381665 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet 687bc729a2f34db58b353ba7a0810b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:54894
I20260620 08:14:39.381703 905 tablet_service.cc:1514] Processing DeleteTablet for tablet 687bc729a2f34db58b353ba7a0810b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:35230
I20260620 08:14:39.381739 1163 tablet_replica.cc:330] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: stopping tablet replica
I20260620 08:14:39.381740 1162 tablet_replica.cc:330] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: stopping tablet replica
I20260620 08:14:39.381781 1162 raft_consensus.cc:2229] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:39.381785 1163 raft_consensus.cc:2229] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:39.381827 1163 raft_consensus.cc:2258] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.381827 1162 raft_consensus.cc:2258] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.381794 1164 tablet_replica.cc:330] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: stopping tablet replica
I20260620 08:14:39.381902 1164 raft_consensus.cc:2229] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:14:39.381984 1164 raft_consensus.cc:2258] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.382165 1162 ts_tablet_manager.cc:1905] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.382180 1163 ts_tablet_manager.cc:1905] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.382534 1164 ts_tablet_manager.cc:1905] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.383812 1163 ts_tablet_manager.cc:1918] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:39.383812 1162 ts_tablet_manager.cc:1918] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:39.383864 1163 log.cc:1196] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/687bc729a2f34db58b353ba7a0810b86
I20260620 08:14:39.383864 1162 log.cc:1196] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/687bc729a2f34db58b353ba7a0810b86
I20260620 08:14:39.384052 1164 ts_tablet_manager.cc:1918] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:14:39.384087 1162 ts_tablet_manager.cc:1939] T 687bc729a2f34db58b353ba7a0810b86 P c7e7c6beecf44d23968db22ca2fd9c5c: Deleting consensus metadata
I20260620 08:14:39.384087 1163 ts_tablet_manager.cc:1939] T 687bc729a2f34db58b353ba7a0810b86 P 50bbd457c8624e009bfd30e93ee5552a: Deleting consensus metadata
I20260620 08:14:39.384105 1164 log.cc:1196] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/687bc729a2f34db58b353ba7a0810b86
I20260620 08:14:39.384366 1164 ts_tablet_manager.cc:1939] T 687bc729a2f34db58b353ba7a0810b86 P d3cad2654db24bee9c365a7e8a36dd14: Deleting consensus metadata
I20260620 08:14:39.384392 615 catalog_manager.cc:4841] TS c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077): tablet 687bc729a2f34db58b353ba7a0810b86 (table test-table [id=0d658aebcb11417186703616ccea8d11]) successfully deleted
I20260620 08:14:39.384456 615 catalog_manager.cc:4841] TS 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309): tablet 687bc729a2f34db58b353ba7a0810b86 (table test-table [id=0d658aebcb11417186703616ccea8d11]) successfully deleted
I20260620 08:14:39.384622 628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:14:39.384835 628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:14:39.385159 612 catalog_manager.cc:4841] TS d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271): tablet 687bc729a2f34db58b353ba7a0810b86 (table test-table [id=0d658aebcb11417186703616ccea8d11]) successfully deleted
I20260620 08:14:39.386031 765 tablet_service.cc:1467] Processing CreateTablet for tablet f94436b09aee409fb4c8b308dd93564c (DEFAULT_TABLE table=test-table [id=924865bf96784aa1978be7ca0ccb1690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:39.386055 1045 tablet_service.cc:1467] Processing CreateTablet for tablet f94436b09aee409fb4c8b308dd93564c (DEFAULT_TABLE table=test-table [id=924865bf96784aa1978be7ca0ccb1690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:39.386090 905 tablet_service.cc:1467] Processing CreateTablet for tablet f94436b09aee409fb4c8b308dd93564c (DEFAULT_TABLE table=test-table [id=924865bf96784aa1978be7ca0ccb1690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:14:39.386142 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94436b09aee409fb4c8b308dd93564c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.386158 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94436b09aee409fb4c8b308dd93564c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.386189 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f94436b09aee409fb4c8b308dd93564c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:39.387295 1139 tablet_bootstrap.cc:492] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c: Bootstrap starting.
I20260620 08:14:39.387341 1140 tablet_bootstrap.cc:492] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: Bootstrap starting.
I20260620 08:14:39.387560 1139 tablet_bootstrap.cc:654] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.387655 1140 tablet_bootstrap.cc:654] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.387775 1143 tablet_bootstrap.cc:492] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: Bootstrap starting.
I20260620 08:14:39.388129 1143 tablet_bootstrap.cc:654] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:39.388366 1140 tablet_bootstrap.cc:492] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: No bootstrap required, opened a new log
I20260620 08:14:39.388437 1140 ts_tablet_manager.cc:1397] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:39.388554 1140 raft_consensus.cc:348] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.388641 1140 raft_consensus.cc:374] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.388664 1140 raft_consensus.cc:729] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bbd457c8624e009bfd30e93ee5552a, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.388715 1140 consensus_queue.cc:260] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.388757 1139 tablet_bootstrap.cc:492] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c: No bootstrap required, opened a new log
I20260620 08:14:39.388806 1139 ts_tablet_manager.cc:1397] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:14:39.388821 1140 ts_tablet_manager.cc:1428] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:39.388921 1139 raft_consensus.cc:348] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.389011 1139 raft_consensus.cc:374] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.389039 1139 raft_consensus.cc:729] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.389088 1139 consensus_queue.cc:260] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.389143 1150 raft_consensus.cc:482] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:14:39.389185 1139 ts_tablet_manager.cc:1428] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:39.389210 1150 raft_consensus.cc:504] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.389338 1150 leader_election.cc:288] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271), 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309)
I20260620 08:14:39.389495 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94436b09aee409fb4c8b308dd93564c" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a" is_pre_election: true
I20260620 08:14:39.389572 1065 raft_consensus.cc:2454] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 0.
I20260620 08:14:39.389709 1143 tablet_bootstrap.cc:492] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: No bootstrap required, opened a new log
I20260620 08:14:39.389757 1143 ts_tablet_manager.cc:1397] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:14:39.389928 1143 raft_consensus.cc:348] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.390007 1143 raft_consensus.cc:374] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:39.390036 1143 raft_consensus.cc:729] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cad2654db24bee9c365a7e8a36dd14, State: Initialized, Role: FOLLOWER
I20260620 08:14:39.390084 1143 consensus_queue.cc:260] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.390194 1143 ts_tablet_manager.cc:1428] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:39.390323 719 leader_election.cc:302] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bbd457c8624e009bfd30e93ee5552a, c7e7c6beecf44d23968db22ca2fd9c5c; no voters:
I20260620 08:14:39.390491 1150 raft_consensus.cc:2781] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:14:39.390543 1150 raft_consensus.cc:482] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:14:39.390566 1150 raft_consensus.cc:3037] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.390837 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94436b09aee409fb4c8b308dd93564c" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14" is_pre_election: true
I20260620 08:14:39.390906 925 raft_consensus.cc:2454] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 0.
I20260620 08:14:39.391301 1150 raft_consensus.cc:504] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.391424 1150 leader_election.cc:288] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 election: Requested vote from peers d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271), 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309)
I20260620 08:14:39.391515 925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94436b09aee409fb4c8b308dd93564c" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cad2654db24bee9c365a7e8a36dd14"
I20260620 08:14:39.391572 925 raft_consensus.cc:3037] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.391551 1065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f94436b09aee409fb4c8b308dd93564c" candidate_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50bbd457c8624e009bfd30e93ee5552a"
I20260620 08:14:39.391616 1065 raft_consensus.cc:3037] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:39.392125 1065 raft_consensus.cc:2454] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 1.
I20260620 08:14:39.392230 925 raft_consensus.cc:2454] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e7c6beecf44d23968db22ca2fd9c5c in term 1.
I20260620 08:14:39.392288 719 leader_election.cc:302] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50bbd457c8624e009bfd30e93ee5552a, c7e7c6beecf44d23968db22ca2fd9c5c; no voters:
I20260620 08:14:39.392445 1150 raft_consensus.cc:2781] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:39.392499 1150 raft_consensus.cc:686] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [term 1 LEADER]: Becoming Leader. State: Replica: c7e7c6beecf44d23968db22ca2fd9c5c, State: Running, Role: LEADER
I20260620 08:14:39.392545 1150 consensus_queue.cc:237] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } }
I20260620 08:14:39.392916 628 catalog_manager.cc:5495] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c reported cstate change: term changed from 0 to 1, leader changed from <none> to c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129). New cstate: current_term: 1 leader_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7e7c6beecf44d23968db22ca2fd9c5c" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 } health_report { overall_health: UNKNOWN } } }
I20260620 08:14:39.400585 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.398092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.24.249.129:33077)
0620 08:14:39.398130 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.398132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.398184 (+ 52us) server_negotiation.cc:407] Connection header received
0620 08:14:39.398234 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.398235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.398243 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.398282 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:39.398398 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.398851 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.399889 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.400161 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.400223 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.400383 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.400410 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.400449 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.400501 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.400518 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:14:39.404603 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.402580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:39337 (local address 127.24.249.129:33077)
0620 08:14:39.402611 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:39.402613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:39.402620 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:39.402640 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:39.402641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:39.402649 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:39.402694 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:14:39.402796 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.403245 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.403953 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.404137 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.404226 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.404357 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.404391 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.404449 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:39.404506 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 08:14:39.404523 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 08:14:39.404610 1192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.402476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:33077 (local address 127.24.249.129:39337)
0620 08:14:39.402537 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:39.402549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:39.402592 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:39.402715 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:39.402716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:39.402717 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 08:14:39.402749 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.402750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.403323 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.403324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.403868 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:39.403870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:39.404225 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:39.404226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:39.404275 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:39.404280 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 08:14:39.404449 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 08:14:39.404451 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:39.404504 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 08:14:39.404545 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 08:14:39.406513 925 raft_consensus.cc:1261] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Refusing update from remote peer c7e7c6beecf44d23968db22ca2fd9c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.406513 1065 raft_consensus.cc:1261] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Refusing update from remote peer c7e7c6beecf44d23968db22ca2fd9c5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:14:39.406738 1145 consensus_queue.cc:1035] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cad2654db24bee9c365a7e8a36dd14" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.406838 1150 consensus_queue.cc:1035] T f94436b09aee409fb4c8b308dd93564c P c7e7c6beecf44d23968db22ca2fd9c5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bbd457c8624e009bfd30e93ee5552a" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:14:39.416837 626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:39.416956 626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:14:39.417618 626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc25077815224057ac32edd836740f6c: Sending DeleteTablet for 3 replicas of tablet f94436b09aee409fb4c8b308dd93564c
I20260620 08:14:39.417840 905 tablet_service.cc:1514] Processing DeleteTablet for tablet f94436b09aee409fb4c8b308dd93564c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:35230
I20260620 08:14:39.417925 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet f94436b09aee409fb4c8b308dd93564c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:14:39 UTC) from {username='oryx', principal='oryx/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:54894
I20260620 08:14:39.418030 1164 tablet_replica.cc:330] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: stopping tablet replica
I20260620 08:14:39.418082 1164 raft_consensus.cc:2229] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:39.418125 1164 raft_consensus.cc:2258] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.418200 1163 tablet_replica.cc:330] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: stopping tablet replica
I20260620 08:14:39.418252 1163 raft_consensus.cc:2229] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:14:39.418291 1163 raft_consensus.cc:2258] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:14:39.418430 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 700
I20260620 08:14:39.418460 1164 ts_tablet_manager.cc:1905] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.419677 1163 ts_tablet_manager.cc:1905] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:14:39.420212 1164 ts_tablet_manager.cc:1918] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:39.420262 1164 log.cc:1196] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/f94436b09aee409fb4c8b308dd93564c
I20260620 08:14:39.420475 1164 ts_tablet_manager.cc:1939] T f94436b09aee409fb4c8b308dd93564c P d3cad2654db24bee9c365a7e8a36dd14: Deleting consensus metadata
I20260620 08:14:39.421129 1163 ts_tablet_manager.cc:1918] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 08:14:39.421193 1163 log.cc:1196] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/f94436b09aee409fb4c8b308dd93564c
I20260620 08:14:39.421403 1163 ts_tablet_manager.cc:1939] T f94436b09aee409fb4c8b308dd93564c P 50bbd457c8624e009bfd30e93ee5552a: Deleting consensus metadata
I20260620 08:14:39.421478 612 catalog_manager.cc:4841] TS d3cad2654db24bee9c365a7e8a36dd14 (127.24.249.130:46271): tablet f94436b09aee409fb4c8b308dd93564c (table test-table [id=924865bf96784aa1978be7ca0ccb1690]) successfully deleted
I20260620 08:14:39.421717 615 catalog_manager.cc:4841] TS 50bbd457c8624e009bfd30e93ee5552a (127.24.249.131:41309): tablet f94436b09aee409fb4c8b308dd93564c (table test-table [id=924865bf96784aa1978be7ca0ccb1690]) successfully deleted
W20260620 08:14:39.426398 615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:33077 (error 108)
I20260620 08:14:39.426635 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 840
I20260620 08:14:39.427498 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:39.426787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:33077 (local address 127.0.0.1:36284)
0620 08:14:39.427397 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:39.427446 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:33077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":41,"threads_started":1}
W20260620 08:14:39.428310 615 catalog_manager.cc:4584] TS c7e7c6beecf44d23968db22ca2fd9c5c (127.24.249.129:33077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f94436b09aee409fb4c8b308dd93564c: Network error: Client connection negotiation failed: client connection to 127.24.249.129:33077: connect: Connection refused (error 111)
I20260620 08:14:39.432178 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 980
I20260620 08:14:39.438352 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 596
2026-06-20T08:14:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2707 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[1231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[1231](info): set up 2 sockets
Jun 20 08:14:39 dist-test-slave-65w8 krb5kdc[1231](info): commencing operation
krb5kdc: starting...
W20260620 08:14:40.486757 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:40 dist-test-slave-65w8 krb5kdc[1231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:40.616725 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:33265
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:33265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:40.698213 1244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:40.698449 1244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:40.698484 1244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:40.699642 1244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:40.699710 1244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:40.699733 1244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:40.699754 1244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:40.699784 1244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:40.701288 1244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:33265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:33265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:40.701560 1244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:40.701802 1244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:40.706688 1250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:40.706688 1252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:40.706796 1249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:40.707191 1244 server_base.cc:1034] running on GCE node
I20260620 08:14:40.707428 1244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:40.707717 1244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:40.708928 1244 hybrid_clock.cc:648] HybridClock initialized: now 1781943280708919 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T08:14:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1274 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:40 dist-test-slave-65w8 krb5kdc[1257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:40 dist-test-slave-65w8 krb5kdc[1257](info): set up 2 sockets
Jun 20 08:14:40 dist-test-slave-65w8 krb5kdc[1257](info): commencing operation
krb5kdc: starting...
W20260620 08:14:41.766748 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:14:41 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:14:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:14:41.893249 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:35415
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:35415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:41.980794 1270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:41.981035 1270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:41.981073 1270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:41.982440 1270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:41.982506 1270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:41.982532 1270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:41.982564 1270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:41.982591 1270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:41.984153 1270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:35415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:35415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:41.984647 1270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:41.984911 1270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:14:41.990213 1270 server_base.cc:1034] running on GCE node
W20260620 08:14:41.990183 1275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:41.990180 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:41.990183 1278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:41.990948 1270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:41.991217 1270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:41.992386 1270 hybrid_clock.cc:648] HybridClock initialized: now 1781943281992361 us; error 51 us; skew 500 ppm
Jun 20 08:14:41 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943281, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:41.995563 1270 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:41.996268 1270 webserver.cc:460] Webserver started at http://127.24.249.190:35615/ using document root <none> and password file <none>
I20260620 08:14:41.996719 1270 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:41.996800 1270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:41.996953 1270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:41.998163 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "78b833be95994524b9da6229685ba309"
format_stamp: "Formatted at 2026-06-20 08:14:41 on dist-test-slave-65w8"
server_key: "34dfde7b3fe667584a09464da316e9ff"
server_key_iv: "0f2aed435281782650db4e91c81b776f"
server_key_version: "encryptionkey@0"
I20260620 08:14:41.998558 1270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "78b833be95994524b9da6229685ba309"
format_stamp: "Formatted at 2026-06-20 08:14:41 on dist-test-slave-65w8"
server_key: "34dfde7b3fe667584a09464da316e9ff"
server_key_iv: "0f2aed435281782650db4e91c81b776f"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.000072 1270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:42.000985 1285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.001260 1270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:42.001376 1270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "78b833be95994524b9da6229685ba309"
format_stamp: "Formatted at 2026-06-20 08:14:41 on dist-test-slave-65w8"
server_key: "34dfde7b3fe667584a09464da316e9ff"
server_key_iv: "0f2aed435281782650db4e91c81b776f"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.001463 1270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:42.023874 1270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:42.026403 1270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:42.026597 1270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:42.031116 1270 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:35415
I20260620 08:14:42.031162 1337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:35415 every 8 connection(s)
I20260620 08:14:42.031517 1270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:42.032384 1338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:42.034754 1338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Bootstrap starting.
I20260620 08:14:42.035342 1338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:42.035683 1338 log.cc:824] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:42.036464 1338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: No bootstrap required, opened a new log
I20260620 08:14:42.038616 1338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.038798 1338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:42.038848 1338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b833be95994524b9da6229685ba309, State: Initialized, Role: FOLLOWER
I20260620 08:14:42.039002 1338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.039088 1338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:42.039125 1338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:42.039178 1338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:42.039326 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 1270
I20260620 08:14:42.039446 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:42.039639 25574 external_mini_cluster.cc:1428] Setting key 1ef5f45115cc4d7260236c67893cc3d5
I20260620 08:14:42.039901 1338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.040030 1338 leader_election.cc:302] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78b833be95994524b9da6229685ba309; no voters:
I20260620 08:14:42.040205 1338 leader_election.cc:288] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:42.040498 1342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:42.040534 1338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:42.040618 1342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 LEADER]: Becoming Leader. State: Replica: 78b833be95994524b9da6229685ba309, State: Running, Role: LEADER
I20260620 08:14:42.040702 1342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.041033 1345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78b833be95994524b9da6229685ba309. Latest consensus state: current_term: 1 leader_uuid: "78b833be95994524b9da6229685ba309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } } }
I20260620 08:14:42.041129 1345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:42.041376 1352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:42.041836 1344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78b833be95994524b9da6229685ba309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } } }
I20260620 08:14:42.041949 1344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:42.041975 1352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:42.046802 1360 mvcc.cc:205] Tried to move back new op lower bound from 7298839683247357952 to 7298839683239059456. Current Snapshot: MvccSnapshot[applied={T|T < 7298839683247357952}]
I20260620 08:14:42.047626 1352 catalog_manager.cc:1347] Generated new cluster ID: 8598477392bb4ab7884394a97ac603b3
I20260620 08:14:42.047688 1352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:42.051069 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.040170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.24.249.190:35415)
0620 08:14:42.040411 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:42.040420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:42.040448 (+ 28us) server_negotiation.cc:407] Connection header received
0620 08:14:42.040587 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:42.040595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:42.040658 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:42.040870 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:42.041692 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.042830 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.043645 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.044053 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.044189 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.046983 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:42.046987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:42.046988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:42.046999 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:42.049464 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.049891 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.049895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.049895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.049988 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.050201 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.050202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.050202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.050441 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:42.050603 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:42.050713 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 08:14:42.050874 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":139,"threads_started":1}
I20260620 08:14:42.065343 1352 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:42.066252 1352 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:42.071092 1352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Generated new TSK 0
I20260620 08:14:42.071300 1352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:14:42.101830 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:35415
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:42.185305 1364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:42.185541 1364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:42.185580 1364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:42.186985 1364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:42.187047 1364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:42.187098 1364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:42.188513 1364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:42.188766 1364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:42.188982 1364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:42.193892 1370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.193967 1372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.193882 1369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:42.194208 1364 server_base.cc:1034] running on GCE node
I20260620 08:14:42.194456 1364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:42.194692 1364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:42.195853 1364 hybrid_clock.cc:648] HybridClock initialized: now 1781943282195840 us; error 28 us; skew 500 ppm
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:42.198364 1364 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:14:42.198818 1364 webserver.cc:460] Webserver started at http://127.24.249.129:38775/ using document root <none> and password file <none>
I20260620 08:14:42.198997 1364 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:42.199035 1364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:42.199124 1364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:42.200156 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "81b99fa8f908437a8de3093f22ac5665"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "81ac9b8f33da15478a8fcdc438ee32f3"
server_key_iv: "a5597d2618bef62bce5a45e5594049f2"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.200464 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "81b99fa8f908437a8de3093f22ac5665"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "81ac9b8f33da15478a8fcdc438ee32f3"
server_key_iv: "a5597d2618bef62bce5a45e5594049f2"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.202004 1364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:14:42.202725 1379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.202901 1364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:14:42.202982 1364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "81b99fa8f908437a8de3093f22ac5665"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "81ac9b8f33da15478a8fcdc438ee32f3"
server_key_iv: "a5597d2618bef62bce5a45e5594049f2"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.203037 1364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:42.242624 1364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:42.245151 1364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:42.245275 1364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:42.245471 1364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:42.245824 1364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:42.245879 1364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.245931 1364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:42.245958 1364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.251713 1364 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:44301
I20260620 08:14:42.251799 1492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:44301 every 8 connection(s)
I20260620 08:14:42.252125 1364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:42.257794 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 1364
I20260620 08:14:42.257939 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:42.258185 25574 external_mini_cluster.cc:1428] Setting key ab86b1a519f03f6da0a5e7ee12c418d9
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:42.262467 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.252976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:43859 (local address 127.24.249.190:35415)
0620 08:14:42.253059 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:42.253061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:42.253300 (+ 239us) server_negotiation.cc:407] Connection header received
0620 08:14:42.253744 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:42.253745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:42.253760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:42.253825 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:42.254801 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.255320 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.256336 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.256538 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.256597 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.259010 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:42.259011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:42.259011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:42.259023 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:42.260558 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.261066 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.261067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.261067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.261152 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.261499 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.261500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.261500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.261631 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:42.261970 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:42.262282 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 08:14:42.262375 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:14:42.262740 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.252932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:43859)
0620 08:14:42.253143 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:42.253187 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:42.253660 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:42.253932 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:42.253939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:42.254504 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:42.254695 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.254699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.255411 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.255412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.256251 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.256254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.256618 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.256619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.256717 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.257198 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:42.257206 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:42.258880 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:42.260642 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:42.260648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:42.260651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:42.260972 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:42.261248 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:42.261253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:42.261254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:42.261407 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:42.261960 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:42.261967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:42.262122 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 08:14:42.262258 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 08:14:42.262420 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260620 08:14:42.263353 1493 heartbeater.cc:347] Connected to a master server at 127.24.249.190:35415
I20260620 08:14:42.263444 1493 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:42.263670 1493 heartbeater.cc:511] Master 127.24.249.190:35415 requested a full tablet report, sending...
I20260620 08:14:42.264322 1302 ts_manager.cc:194] Registered new tserver with Master: 81b99fa8f908437a8de3093f22ac5665 (127.24.249.129:44301)
I20260620 08:14:42.265331 1302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:43859
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:14:42.285460 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:35415
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:42.365556 1498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:42.365783 1498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:42.365808 1498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:42.367076 1498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:42.367129 1498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:42.367157 1498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:42.368556 1498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:42.368765 1498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:42.368939 1498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:42.374004 1506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.374112 1504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.374010 1503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:42.374471 1498 server_base.cc:1034] running on GCE node
I20260620 08:14:42.374697 1498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:42.375028 1498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:42.376245 1498 hybrid_clock.cc:648] HybridClock initialized: now 1781943282376211 us; error 47 us; skew 500 ppm
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:42.379127 1498 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:14:42.379894 1498 webserver.cc:460] Webserver started at http://127.24.249.130:44735/ using document root <none> and password file <none>
I20260620 08:14:42.380189 1498 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:42.380266 1498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:42.380417 1498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:42.381500 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "adbdd3e5739944d48ddada4fd6933e83"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "69d160258ade227506075248b54b953a"
server_key_iv: "c4fb19062289dbfde6367deac4c09955"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.381965 1498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "adbdd3e5739944d48ddada4fd6933e83"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "69d160258ade227506075248b54b953a"
server_key_iv: "c4fb19062289dbfde6367deac4c09955"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.383534 1498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:42.384433 1513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.384686 1498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:42.384817 1498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "adbdd3e5739944d48ddada4fd6933e83"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "69d160258ade227506075248b54b953a"
server_key_iv: "c4fb19062289dbfde6367deac4c09955"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.384902 1498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:42.394621 1498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:42.396548 1498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:42.396718 1498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:42.396931 1498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:42.397253 1498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:42.397310 1498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.397351 1498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:42.397378 1498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.405030 1498 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:33747
I20260620 08:14:42.405189 1626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:33747 every 8 connection(s)
I20260620 08:14:42.405478 1498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:42.410681 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 1498
I20260620 08:14:42.410805 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:42.411018 25574 external_mini_cluster.cc:1428] Setting key 43fb4a0fa0f4085f2c2d78629f61bf10
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:42.416765 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.406326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:43061 (local address 127.24.249.190:35415)
0620 08:14:42.406400 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:42.406403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:42.406705 (+ 302us) server_negotiation.cc:407] Connection header received
0620 08:14:42.407276 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:42.407278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:42.407293 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:42.407348 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:42.408745 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.409352 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.410877 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.411170 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.411217 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.413803 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:42.413804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:42.413804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:42.413812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:42.415333 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.415774 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.415775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.415776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.415838 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.416095 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.416096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.416097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.416219 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:42.416295 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:42.416572 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 08:14:42.416660 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:14:42.416955 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.406308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:43061)
0620 08:14:42.406526 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:42.406579 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:42.407119 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:42.407457 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:42.407461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:42.408365 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:42.408647 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.408654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.409436 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.409438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.410756 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.410760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.411315 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.411316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.411466 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.412033 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:42.412041 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:42.413677 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:42.415436 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:42.415442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:42.415444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:42.415671 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:42.415905 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:42.415906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:42.415907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:42.416017 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:42.416286 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:42.416293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:42.416438 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 08:14:42.416552 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:14:42.416709 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
I20260620 08:14:42.417467 1627 heartbeater.cc:347] Connected to a master server at 127.24.249.190:35415
I20260620 08:14:42.417549 1627 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:42.417732 1627 heartbeater.cc:511] Master 127.24.249.190:35415 requested a full tablet report, sending...
I20260620 08:14:42.418210 1302 ts_manager.cc:194] Registered new tserver with Master: adbdd3e5739944d48ddada4fd6933e83 (127.24.249.130:33747)
I20260620 08:14:42.418771 1302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:43061
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:14:42.439448 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:35415
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:42.524982 1632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:42.525353 1632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:42.525398 1632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:42.526865 1632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:42.526917 1632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:42.526947 1632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:42.528478 1632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:42.528738 1632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:42.529016 1632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:42.534545 1637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.534554 1638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:42.534570 1632 server_base.cc:1034] running on GCE node
W20260620 08:14:42.534554 1640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:42.535115 1632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:42.535408 1632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:42.536576 1632 hybrid_clock.cc:648] HybridClock initialized: now 1781943282536464 us; error 126 us; skew 500 ppm
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:42.539896 1632 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:14:42.541034 1632 webserver.cc:460] Webserver started at http://127.24.249.131:42177/ using document root <none> and password file <none>
I20260620 08:14:42.541476 1632 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:42.541563 1632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:42.541738 1632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:42.543460 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "8a03c5cefe1b4ff19ecb2ac1bd6bcb0c"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "f199975ff545bd74776a29f3624d01d4"
server_key_iv: "5baae4ad59b78d05e1bd6023e5d0ce12"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.543908 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "8a03c5cefe1b4ff19ecb2ac1bd6bcb0c"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "f199975ff545bd74776a29f3624d01d4"
server_key_iv: "5baae4ad59b78d05e1bd6023e5d0ce12"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.545725 1632 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 08:14:42.546720 1647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.547076 1632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:42.547283 1632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "8a03c5cefe1b4ff19ecb2ac1bd6bcb0c"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "f199975ff545bd74776a29f3624d01d4"
server_key_iv: "5baae4ad59b78d05e1bd6023e5d0ce12"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.547411 1632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:42.561177 1632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:42.563661 1632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:42.563843 1632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:42.564098 1632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:42.564409 1632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:42.564464 1632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.564513 1632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:42.564543 1632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.572057 1632 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:33349
I20260620 08:14:42.572096 1760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:33349 every 8 connection(s)
I20260620 08:14:42.572459 1632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:42.575588 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 1632
I20260620 08:14:42.575724 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:42.575908 25574 external_mini_cluster.cc:1428] Setting key dbb3bd75df6f975e5d4003d948672bfe
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:14:42.584043 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.573543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:43259 (local address 127.24.249.190:35415)
0620 08:14:42.573670 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:42.573673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:42.573685 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:42.574240 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:42.574242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:42.574255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:42.574300 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:42.575239 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.575786 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.577486 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.577699 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.577789 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.580561 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:42.580563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:42.580564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:42.580572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:42.582362 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.582888 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.582890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.582890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.582979 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:42.583311 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:42.583313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:42.583313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:42.583441 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:42.583523 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:42.583851 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 08:14:42.583942 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:14:42.584295 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.573327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:43259)
0620 08:14:42.573485 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:42.573529 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:42.574125 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:42.574366 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:42.574370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:42.574940 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:42.575132 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.575136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.575863 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.575865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.577359 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.577362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.577799 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.577800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.577973 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.578595 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:42.578608 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:42.580449 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:42.582485 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:42.582496 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:42.582499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:42.582775 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:42.583095 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:14:42.583096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:14:42.583096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:14:42.583221 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:14:42.583513 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:42.583520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:42.583686 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 08:14:42.583824 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 08:14:42.584014 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260620 08:14:42.584877 1761 heartbeater.cc:347] Connected to a master server at 127.24.249.190:35415
I20260620 08:14:42.584975 1761 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:42.585124 1761 heartbeater.cc:511] Master 127.24.249.190:35415 requested a full tablet report, sending...
I20260620 08:14:42.585593 1302 ts_manager.cc:194] Registered new tserver with Master: 8a03c5cefe1b4ff19ecb2ac1bd6bcb0c (127.24.249.131:33349)
I20260620 08:14:42.586097 1302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:43259
I20260620 08:14:42.588483 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab.
Entry for principal oryx/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab.
WARNING: no policy specified for oryx/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab.
Entry for principal oryx/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab.
WARNING: no policy specified for oryx/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab.
Entry for principal oryx/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab.
I20260620 08:14:42.673542 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 1364
I20260620 08:14:42.678535 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 1498
I20260620 08:14:42.683293 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 1632
I20260620 08:14:42.687836 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 1270
I20260620 08:14:42.693041 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:35415
--webserver_interface=127.24.249.190
--webserver_port=35615
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:35415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:14:42.776579 1770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:42.776814 1770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:42.776852 1770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:42.778249 1770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:42.778342 1770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:42.778373 1770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:42.778406 1770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:42.778435 1770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:42.780071 1770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:35415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:35415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=35615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:42.780393 1770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:42.780632 1770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:42.785663 1778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.785748 1775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.785681 1776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:42.786003 1770 server_base.cc:1034] running on GCE node
I20260620 08:14:42.786309 1770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:42.786612 1770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:42.787792 1770 hybrid_clock.cc:648] HybridClock initialized: now 1781943282787771 us; error 29 us; skew 500 ppm
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:42.790397 1770 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:14:42.790871 1770 webserver.cc:460] Webserver started at http://127.24.249.190:35615/ using document root <none> and password file <none>
I20260620 08:14:42.791124 1770 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:42.791205 1770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:42.792546 1770 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:42.793207 1785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.793375 1770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.793475 1770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "78b833be95994524b9da6229685ba309"
format_stamp: "Formatted at 2026-06-20 08:14:41 on dist-test-slave-65w8"
server_key: "34dfde7b3fe667584a09464da316e9ff"
server_key_iv: "0f2aed435281782650db4e91c81b776f"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.793849 1770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:42.812517 1770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:42.814783 1770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:42.814981 1770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:42.819427 1770 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:35415
I20260620 08:14:42.819480 1837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:35415 every 8 connection(s)
I20260620 08:14:42.819892 1770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:42.821579 1838 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:14:42.822365 1838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Bootstrap starting.
I20260620 08:14:42.828473 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 1770
I20260620 08:14:42.828840 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:44301
--local_ip_for_outbound_sockets=127.24.249.129
--tserver_master_addrs=127.24.249.190:35415
--webserver_port=38775
--webserver_interface=127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:14:42.830891 1838 log.cc:824] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:42.832872 1838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:14:42.833194 1838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Bootstrap complete.
I20260620 08:14:42.835570 1838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.835829 1838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b833be95994524b9da6229685ba309, State: Initialized, Role: FOLLOWER
I20260620 08:14:42.836025 1838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.836115 1838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:42.836169 1838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:42.836218 1838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:14:42.837224 1838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.837343 1838 leader_election.cc:302] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78b833be95994524b9da6229685ba309; no voters:
I20260620 08:14:42.837538 1838 leader_election.cc:288] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:14:42.837574 1842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:14:42.837689 1842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [term 2 LEADER]: Becoming Leader. State: Replica: 78b833be95994524b9da6229685ba309, State: Running, Role: LEADER
I20260620 08:14:42.837824 1838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:42.837894 1842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } }
I20260620 08:14:42.838271 1843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "78b833be95994524b9da6229685ba309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } } }
I20260620 08:14:42.838286 1844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78b833be95994524b9da6229685ba309. Latest consensus state: current_term: 2 leader_uuid: "78b833be95994524b9da6229685ba309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b833be95994524b9da6229685ba309" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 35415 } } }
I20260620 08:14:42.838382 1843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:42.838389 1844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:42.839726 1858 catalog_manager.cc:1259] Loaded cluster ID: 8598477392bb4ab7884394a97ac603b3
I20260620 08:14:42.839776 1858 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: loading cluster ID for follower catalog manager: success
I20260620 08:14:42.841012 1858 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: acquiring CA information for follower catalog manager: success
I20260620 08:14:42.841380 1858 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:14:42.841454 1850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:42.841640 1850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:42.841701 1850 catalog_manager.cc:1259] Loaded cluster ID: 8598477392bb4ab7884394a97ac603b3
I20260620 08:14:42.841722 1850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:42.841907 1850 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:42.841969 1850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78b833be95994524b9da6229685ba309: Loaded TSK: 0
I20260620 08:14:42.842079 1850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:14:42.913156 1840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:42.913407 1840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:42.913444 1840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:42.914740 1840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:42.914821 1840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:42.914870 1840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:42.916363 1840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.129.keytab
--principal=oryx/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:44301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=38775
--tserver_master_addrs=127.24.249.190:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:42.916652 1840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:42.916891 1840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:42.921955 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:42.921965 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:42.922132 1840 server_base.cc:1034] running on GCE node
W20260620 08:14:42.922302 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:42.922549 1840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:42.922821 1840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:42.924007 1840 hybrid_clock.cc:648] HybridClock initialized: now 1781943282923991 us; error 31 us; skew 500 ppm
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943282, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:42.926865 1840 init.cc:377] Logged in from keytab as oryx/127.24.249.129@KRBTEST.COM (short username oryx)
I20260620 08:14:42.927387 1840 webserver.cc:460] Webserver started at http://127.24.249.129:38775/ using document root <none> and password file <none>
I20260620 08:14:42.927625 1840 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:42.927711 1840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:42.929129 1840 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:42.929797 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.930058 1840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:42.930189 1840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "81b99fa8f908437a8de3093f22ac5665"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "81ac9b8f33da15478a8fcdc438ee32f3"
server_key_iv: "a5597d2618bef62bce5a45e5594049f2"
server_key_version: "encryptionkey@0"
I20260620 08:14:42.930541 1840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:42.965758 1840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:42.968009 1840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:42.968178 1840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:42.968451 1840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:42.968776 1840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:42.968811 1840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.968833 1840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:42.968845 1840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:42.975113 1840 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:44301
I20260620 08:14:42.975155 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:44301 every 8 connection(s)
I20260620 08:14:42.975499 1840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:42.984292 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.976230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:59749)
0620 08:14:42.976387 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:42.976431 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:42.976899 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:42.977326 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:42.977330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:42.978017 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:42.978223 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.978228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.979410 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.979411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.980225 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.980227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.980872 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.980873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.980978 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.981570 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:42.981578 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:42.984139 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
W20260620 08:14:42.984501 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:42.976314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:59749 (local address 127.24.249.190:35415)
0620 08:14:42.976508 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:42.976516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:42.976555 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:14:42.977064 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:42.977069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:42.977132 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:42.977249 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:42.978382 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.979296 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.980334 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.980773 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.980873 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.984291 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:59749: BlockingRecv error: recv got EOF from 127.24.249.129:59749 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
W20260620 08:14:42.984572 1988 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:35415 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.249.190:35415: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
I20260620 08:14:42.984863 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 1840
I20260620 08:14:42.985229 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:33747
--local_ip_for_outbound_sockets=127.24.249.130
--tserver_master_addrs=127.24.249.190:35415
--webserver_port=44735
--webserver_interface=127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:42.989719 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.984807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:38639)
0620 08:14:42.984930 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:42.984939 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:42.985006 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:42.985165 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:42.985166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:42.985282 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:42.985353 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.985355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.986053 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.986054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.986863 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.986866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.987355 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.987356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.987408 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.987886 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:42.987891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:42.989670 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 08:14:42.990163 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:42.984881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:38639 (local address 127.24.249.190:35415)
0620 08:14:42.984945 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:42.984947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:42.985036 (+ 89us) server_negotiation.cc:407] Connection header received
0620 08:14:42.985063 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:42.985064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:42.985079 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:42.985150 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:42.985440 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.985979 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.986945 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.987276 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.987355 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.989904 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:38639: BlockingRecv error: recv got EOF from 127.24.249.129:38639 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:42 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:42.995219 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:42.990044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:43065)
0620 08:14:42.990124 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:42.990137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:42.990203 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:42.990417 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:42.990418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:42.990557 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:42.990589 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.990632 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.991653 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.991654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.992144 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:42.992146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.992699 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.992700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:42.992745 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.993248 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:42.993252 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:42.995174 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 08:14:42.995368 1988 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:14:42.995379 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:42.990179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:43065 (local address 127.24.249.190:35415)
0620 08:14:42.990282 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:42.990285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:42.990293 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:14:42.990317 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:42.990317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:42.990329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:42.990399 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:42.990719 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.991519 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.992242 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:42.992575 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:42.992655 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:42.995296 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:43065: BlockingRecv error: recv got EOF from 127.24.249.129:43065 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":101,"threads_started":1}
W20260620 08:14:43.069659 1992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:43.069950 1992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:43.069998 1992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:43.071175 1992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:43.071213 1992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:43.071239 1992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:43.072616 1992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.130.keytab
--principal=oryx/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:33747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=44735
--tserver_master_addrs=127.24.249.190:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:43.072801 1992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:43.072983 1992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:43.078019 1998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:43.078034 1999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:43.078020 2001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:43.078312 1992 server_base.cc:1034] running on GCE node
I20260620 08:14:43.078534 1992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:43.078804 1992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:43.079957 1992 hybrid_clock.cc:648] HybridClock initialized: now 1781943283079929 us; error 50 us; skew 500 ppm
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943283, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:43.082516 1992 init.cc:377] Logged in from keytab as oryx/127.24.249.130@KRBTEST.COM (short username oryx)
I20260620 08:14:43.082954 1992 webserver.cc:460] Webserver started at http://127.24.249.130:44735/ using document root <none> and password file <none>
I20260620 08:14:43.083204 1992 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:43.083285 1992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:43.084740 1992 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:43.085393 2008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:43.085580 1992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:43.085709 1992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "adbdd3e5739944d48ddada4fd6933e83"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "69d160258ade227506075248b54b953a"
server_key_iv: "c4fb19062289dbfde6367deac4c09955"
server_key_version: "encryptionkey@0"
I20260620 08:14:43.086108 1992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:43.099764 1992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:43.102056 1992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:43.102223 1992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:43.102475 1992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:43.102793 1992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:43.102850 1992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:43.102891 1992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:43.102914 1992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:43.108728 1992 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:33747
I20260620 08:14:43.108803 2121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:33747 every 8 connection(s)
I20260620 08:14:43.109151 1992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:43.110953 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 1992
I20260620 08:14:43.111308 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:33349
--local_ip_for_outbound_sockets=127.24.249.131
--tserver_master_addrs=127.24.249.190:35415
--webserver_port=42177
--webserver_interface=127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:14:43.117566 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:43.109961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:58227 (local address 127.24.249.190:35415)
0620 08:14:43.110019 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.110021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.110196 (+ 175us) server_negotiation.cc:407] Connection header received
0620 08:14:43.110697 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.110698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.110712 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.110777 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.112146 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.112684 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.113740 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.114003 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.114093 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.117503 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:58227: BlockingRecv error: recv got EOF from 127.24.249.130:58227 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:14:43.117688 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.109946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:58227)
0620 08:14:43.110089 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:43.110126 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:43.110579 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:43.110895 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:43.110899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:43.111750 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:43.112034 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.112040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.112832 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.112833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.113635 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.113637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.114104 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.114105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.114226 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.114830 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:43.114839 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:43.117419 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
W20260620 08:14:43.118074 2122 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:35415 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.249.190:35415: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:43.123078 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.118373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:50433)
0620 08:14:43.118410 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:43.118423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:43.118468 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:43.118609 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:43.118610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:43.118755 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:43.118803 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.118804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.119514 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.119515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.120073 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.120075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.120497 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.120498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.120579 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.121074 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:43.121079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:43.123006 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 08:14:43.123236 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:43.118440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:50433 (local address 127.24.249.190:35415)
0620 08:14:43.118484 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.118486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.118495 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:43.118524 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.118524 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.118534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.118588 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.118896 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.119426 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.120184 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.120403 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.120497 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.123148 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:50433: BlockingRecv error: recv got EOF from 127.24.249.130:50433 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:43.128012 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.123344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:38375)
0620 08:14:43.123426 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:43.123436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:43.123473 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:43.123616 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:43.123617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:43.123753 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:43.123786 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.123788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.124420 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.124421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.125233 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.125234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.125599 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.125600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.125653 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.126147 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:43.126152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:43.127966 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 08:14:43.128167 2122 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:14:43.128221 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:43.123431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:38375 (local address 127.24.249.190:35415)
0620 08:14:43.123489 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.123491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.123500 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:43.123519 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.123519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.123528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.123598 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.123882 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.124344 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.125332 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.125522 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.125586 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.128143 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:38375: BlockingRecv error: recv got EOF from 127.24.249.130:38375 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 08:14:43.198966 2125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:43.199188 2125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:43.199214 2125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:43.200383 2125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:43.200424 2125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:43.200451 2125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:43.201930 2125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/krb5kdc/oryx_127.24.249.131.keytab
--principal=oryx/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:33349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=42177
--tserver_master_addrs=127.24.249.190:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:43.202132 2125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:43.202316 2125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:43.207440 2133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:43.207434 2130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:43.207437 2131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:43.207911 2125 server_base.cc:1034] running on GCE node
I20260620 08:14:43.208171 2125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:43.208407 2125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:43.209579 2125 hybrid_clock.cc:648] HybridClock initialized: now 1781943283209540 us; error 74 us; skew 500 ppm
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943283, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:14:43.212817 2125 init.cc:377] Logged in from keytab as oryx/127.24.249.131@KRBTEST.COM (short username oryx)
I20260620 08:14:43.213392 2125 webserver.cc:460] Webserver started at http://127.24.249.131:42177/ using document root <none> and password file <none>
I20260620 08:14:43.213661 2125 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:43.213752 2125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:43.215422 2125 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:43.216424 2140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:43.216943 2125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:43.217331 2125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "8a03c5cefe1b4ff19ecb2ac1bd6bcb0c"
format_stamp: "Formatted at 2026-06-20 08:14:42 on dist-test-slave-65w8"
server_key: "f199975ff545bd74776a29f3624d01d4"
server_key_iv: "5baae4ad59b78d05e1bd6023e5d0ce12"
server_key_version: "encryptionkey@0"
I20260620 08:14:43.217757 2125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:43.234860 2125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:43.237473 2125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:43.237679 2125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:43.237998 2125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:43.238364 2125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:43.238432 2125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:43.238477 2125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:43.238504 2125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:43.247673 2125 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:33349
I20260620 08:14:43.247733 2253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:33349 every 8 connection(s)
I20260620 08:14:43.248103 2125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:43.256157 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.248876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:46069)
0620 08:14:43.249097 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:43.249173 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:43.249689 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:43.250067 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:43.250071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:43.250676 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:43.250918 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.250923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.251615 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.251616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.252475 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.252477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.252845 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.252846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.252959 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.253524 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:43.253533 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:43.255995 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
W20260620 08:14:43.256188 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:43.248916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:46069 (local address 127.24.249.190:35415)
0620 08:14:43.249002 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.249004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.249287 (+ 283us) server_negotiation.cc:407] Connection header received
0620 08:14:43.249892 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.249894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.249909 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.250002 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.251018 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.251535 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.252557 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.252742 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.252824 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.256125 (+ 3301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:46069: BlockingRecv error: recv got EOF from 127.24.249.131:46069 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 08:14:43.256474 2254 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:35415 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.249.190:35415: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
I20260620 08:14:43.257292 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 2125
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:43.262271 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.256800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:51083)
0620 08:14:43.256875 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:43.256886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:43.256927 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:43.257208 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:43.257209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:43.257322 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:43.257501 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.257503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.258316 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.258317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.259043 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.259045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.259493 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.259495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.259565 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.260032 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:43.260038 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:43.262186 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 08:14:43.262319 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:43.256987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:51083 (local address 127.24.249.190:35415)
0620 08:14:43.257047 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.257049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.257056 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:43.257105 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.257106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.257117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.257186 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.257587 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.258210 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.259121 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.259411 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.259472 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.262253 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:51083: BlockingRecv error: recv got EOF from 127.24.249.131:51083 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:14:43.265938 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.257683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.24.249.190:35415)
0620 08:14:43.257759 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.257761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.257925 (+ 164us) server_negotiation.cc:407] Connection header received
0620 08:14:43.257997 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.257998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.258010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.258075 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.258760 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.259321 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.260456 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.260752 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.260828 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.262329 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:43.262336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:43.262336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:14:43.262351 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:43.264400 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:43.264798 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:43.264802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:43.264802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:43.264909 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:14:43.265131 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:14:43.265132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:14:43.265132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:14:43.265381 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:43.265537 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:43.265658 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 08:14:43.265773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:43.267297 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.262569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:35009)
0620 08:14:43.262609 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:43.262619 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:43.262650 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:43.262807 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:43.262808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:43.262953 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:43.262989 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.262991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.263908 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.263909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.264445 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.264446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.264863 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.264864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.264917 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.265320 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:43.265325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:43.267244 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 08:14:43.267446 2254 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:14:43.267468 1991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:43.262647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:35009 (local address 127.24.249.190:35415)
0620 08:14:43.262686 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.262688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.262697 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:43.262716 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.262717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.262726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.262782 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.263077 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.263815 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.264573 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.264774 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.264863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.267404 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:35009: BlockingRecv error: recv got EOF from 127.24.249.131:35009 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:43 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:44.001268 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:43.995805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:37119)
0620 08:14:43.995985 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:43.995998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:43.996058 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:43.996259 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:43.996260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:43.996428 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:43.996507 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.996510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.997283 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.997284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.997931 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:43.997933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.998333 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.998334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:43.998389 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:43.999060 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:43.999065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:44.001166 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
W20260620 08:14:44.001333 2259 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:43.995951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:37119 (local address 127.24.249.190:35415)
0620 08:14:43.996087 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:43.996089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:43.996099 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:43.996138 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:43.996138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:43.996148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:43.996219 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:43.996618 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.997142 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.998026 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:43.998248 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:43.998314 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:44.001261 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:37119: BlockingRecv error: recv got EOF from 127.24.249.129:37119 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
Jun 20 08:14:44 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:44 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:44.134128 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:44.128595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:41615)
0620 08:14:44.128853 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:44.128866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:44.128966 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:44.129218 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:44.129220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:44.129390 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:44.129436 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:44.129437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.130272 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.130273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:44.130872 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:44.130874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.131271 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.131272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:44.131325 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:44.131897 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:44.131902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:44.133991 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
W20260620 08:14:44.134143 2259 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:44.128697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:41615 (local address 127.24.249.190:35415)
0620 08:14:44.128853 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:44.128855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:44.128945 (+ 90us) server_negotiation.cc:407] Connection header received
0620 08:14:44.129088 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:44.129089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:44.129101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:44.129161 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:44.129541 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.130176 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.130971 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.131169 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.131243 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:44.134085 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:41615: BlockingRecv error: recv got EOF from 127.24.249.130:41615 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Jun 20 08:14:44 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:44 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:44.273128 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:44.267966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:54563)
0620 08:14:44.268160 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:44.268174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:44.268254 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:44.268442 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:44.268443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:44.268606 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:44.268654 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:44.268656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.269395 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.269396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:44.270071 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:44.270072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.270520 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.270521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:44.270604 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:44.271100 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:44.271104 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:44.273061 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":58,"threads_started":1}
W20260620 08:14:44.273288 2259 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:44.268111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:54563 (local address 127.24.249.190:35415)
0620 08:14:44.268207 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:44.268209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:44.268233 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:14:44.268318 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:44.268319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:44.268337 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:44.268388 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:44.268791 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.269309 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.270144 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:44.270422 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:44.270496 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:44.273165 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:54563: BlockingRecv error: recv got EOF from 127.24.249.131:54563 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 08:14:45 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:45 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:45.007623 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:45.001938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:52961)
0620 08:14:45.002098 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:45.002117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:45.002206 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:45.002534 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:45.002536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:45.002772 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:45.002824 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:45.002826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.003610 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.003611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:45.004369 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:45.004371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.004836 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.004837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:45.004885 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:45.005430 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:45.005435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:45.007558 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
W20260620 08:14:45.007838 2263 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:45.002011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:52961 (local address 127.24.249.190:35415)
0620 08:14:45.002147 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:45.002149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:45.002189 (+ 40us) server_negotiation.cc:407] Connection header received
0620 08:14:45.002330 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:45.002331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:45.002344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:45.002450 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:45.002957 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.003483 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.004483 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.004735 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.004800 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:45.007729 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:52961: BlockingRecv error: recv got EOF from 127.24.249.129:52961 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
Jun 20 08:14:45 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:45 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:45.140461 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:45.134730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:43829)
0620 08:14:45.134857 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:45.134876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:45.134938 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:45.135160 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:45.135161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:45.135331 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:45.135382 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:45.135383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.136523 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.136524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:45.137064 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:45.137066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.137572 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.137573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:45.137673 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:45.138289 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:45.138295 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:45.140393 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
W20260620 08:14:45.140671 2263 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:45.134811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:43829 (local address 127.24.249.190:35415)
0620 08:14:45.134897 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:45.134899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:45.134924 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:14:45.135040 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:45.135041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:45.135055 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:45.135140 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:45.135517 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.136444 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.137175 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.137451 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.137515 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:45.140533 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:43829: BlockingRecv error: recv got EOF from 127.24.249.130:43829 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 08:14:45 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:45 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:45.279065 2265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:45.273844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:48049)
0620 08:14:45.274097 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:45.274111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:45.274209 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:45.274358 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:45.274359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:45.274531 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:45.274585 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:45.274587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.275284 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.275285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:45.275823 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:45.275824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.276207 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.276208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:45.276264 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:45.276907 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:45.276912 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:45.278992 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":135,"threads_started":1}
W20260620 08:14:45.279211 2263 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:45.274024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:48049 (local address 127.24.249.190:35415)
0620 08:14:45.274142 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:45.274144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:45.274208 (+ 64us) server_negotiation.cc:407] Connection header received
0620 08:14:45.274271 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:45.274272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:45.274287 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:45.274351 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:45.274676 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.275183 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.275912 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:45.276117 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:45.276207 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:45.279091 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:48049: BlockingRecv error: recv got EOF from 127.24.249.131:48049 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 08:14:46 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:46 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:46.018888 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:46.008261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:34495)
0620 08:14:46.008391 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:46.008409 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:46.008460 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:46.008601 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:46.008602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:46.008767 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:46.008815 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:46.008817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.009520 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.009521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:46.010218 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:46.010221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.010612 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.010613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:46.010672 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:46.016697 (+ 6025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:46.016707 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:46.018804 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
W20260620 08:14:46.019048 2267 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:46.008319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:34495 (local address 127.24.249.190:35415)
0620 08:14:46.008471 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:46.008473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:46.008482 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:46.008505 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:46.008505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:46.008515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:46.008582 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:46.008895 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.009419 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.010320 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.010528 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.010592 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:46.018954 (+ 8362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:34495: BlockingRecv error: recv got EOF from 127.24.249.129:34495 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
Jun 20 08:14:46 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:46 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:46.145967 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:46.140984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:47473)
0620 08:14:46.141172 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:46.141186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:46.141285 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:46.141453 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:46.141454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:46.141636 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:46.141683 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:46.141685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.142433 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.142435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:46.142992 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:46.142994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.143395 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.143396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:46.143457 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:46.143987 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:46.143997 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:46.145840 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
W20260620 08:14:46.146031 2267 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:46.141079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:47473 (local address 127.24.249.190:35415)
0620 08:14:46.141177 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:46.141180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:46.141272 (+ 92us) server_negotiation.cc:407] Connection header received
0620 08:14:46.141360 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:46.141361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:46.141378 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:46.141447 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:46.141772 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.142344 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.143104 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.143303 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.143393 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:46.145969 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:47473: BlockingRecv error: recv got EOF from 127.24.249.130:47473 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 08:14:46 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:46 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:46.284353 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:46.279577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:40759)
0620 08:14:46.279733 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:46.279749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:46.279807 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:46.279958 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:46.279959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:46.280118 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:46.280166 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:46.280168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.280833 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.280834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:46.281371 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:46.281373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.281695 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.281696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:46.281737 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:46.282225 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:46.282230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:46.284293 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260620 08:14:46.284476 2267 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:46.279680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:40759 (local address 127.24.249.190:35415)
0620 08:14:46.279749 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:46.279751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:46.279797 (+ 46us) server_negotiation.cc:407] Connection header received
0620 08:14:46.279865 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:46.279866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:46.279878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:46.279939 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:46.280243 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.280733 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.281449 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:46.281626 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:46.281676 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:46.284379 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:40759: BlockingRecv error: recv got EOF from 127.24.249.131:40759 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:14:47 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:47 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:47.024969 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:47.019497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:52397)
0620 08:14:47.019692 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:47.019703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:47.019767 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:47.019958 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:47.019960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:47.020156 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:47.020211 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:47.020213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.021092 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.021093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:47.021748 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:47.021751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.022204 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.022205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:47.022279 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:47.023001 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:47.023006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:47.024910 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
W20260620 08:14:47.025264 2271 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:47.019636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:52397 (local address 127.24.249.190:35415)
0620 08:14:47.019754 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:47.019756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:47.019765 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:47.019835 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:47.019836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:47.019846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:47.019907 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:47.020350 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.020991 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.021894 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.022088 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.022152 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:47.025180 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:52397: BlockingRecv error: recv got EOF from 127.24.249.129:52397 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":48,"threads_started":1}
Jun 20 08:14:47 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:47 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:47.151253 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:47.146545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:36069)
0620 08:14:47.146738 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:47.146748 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:47.146806 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:47.146940 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:47.146941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:47.147104 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:47.147152 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:47.147154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.147886 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.147887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:47.148431 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:47.148434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.148788 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.148789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:47.148835 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:47.149418 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:47.149423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:47.151194 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
W20260620 08:14:47.151441 2271 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:47.146623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:36069 (local address 127.24.249.190:35415)
0620 08:14:47.146714 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:47.146717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:47.146812 (+ 95us) server_negotiation.cc:407] Connection header received
0620 08:14:47.146852 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:47.146853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:47.146864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:47.146920 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:47.147264 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.147790 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.148516 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.148709 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.148768 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:47.151361 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:36069: BlockingRecv error: recv got EOF from 127.24.249.130:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 08:14:47 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:47 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:47.295647 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:47.285042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:42567)
0620 08:14:47.285200 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:47.285215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:47.285288 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:47.285463 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:47.285464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:47.285636 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:47.285684 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:47.285685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.286452 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.286453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:47.286979 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:47.286980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.287379 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.287380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:47.287444 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:47.293077 (+ 5633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:47.293084 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:47.295549 (+ 2465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
W20260620 08:14:47.295729 2271 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:47.285150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:42567 (local address 127.24.249.190:35415)
0620 08:14:47.285231 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:47.285234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:47.285277 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:14:47.285392 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:47.285393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:47.285406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:47.285458 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:47.285777 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.286352 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.287076 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:47.287290 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:47.287376 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:47.295653 (+ 8277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:42567: BlockingRecv error: recv got EOF from 127.24.249.131:42567 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 08:14:48 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:48 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:48.030730 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:48.025520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:47647)
0620 08:14:48.025671 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:48.025681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:48.025737 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:48.025975 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:48.025976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:48.026169 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:48.026220 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:48.026222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.027064 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.027066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:48.027615 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:48.027617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.028001 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.028002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:48.028068 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:48.028586 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:48.028590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:48.030632 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
W20260620 08:14:48.030803 2275 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:48.025619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:47647 (local address 127.24.249.190:35415)
0620 08:14:48.025792 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:48.025795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:48.025808 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:14:48.025837 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:48.025837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:48.025848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:48.025948 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:48.026318 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.026977 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.027706 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.027911 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.027990 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:48.030715 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:47647: BlockingRecv error: recv got EOF from 127.24.249.129:47647 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
Jun 20 08:14:48 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:48 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:48.156981 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:48.151782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:54017)
0620 08:14:48.151929 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:48.151941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:48.151998 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:48.152143 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:48.152144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:48.152319 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:48.152367 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:48.152368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.153098 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.153099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:48.153793 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:48.153795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.154218 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.154219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:48.154267 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:48.154920 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:48.154925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:48.156910 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
W20260620 08:14:48.157107 2275 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:48.151954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:54017 (local address 127.24.249.190:35415)
0620 08:14:48.152009 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:48.152010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:48.152017 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:14:48.152043 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:48.152044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:48.152054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:48.152110 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:48.152468 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.153002 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.153921 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.154115 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.154194 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:48.157021 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:54017: BlockingRecv error: recv got EOF from 127.24.249.130:54017 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 08:14:48 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:48 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:48.301229 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:48.296253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:44645)
0620 08:14:48.296447 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:48.296465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:48.296527 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:48.296665 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:48.296666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:48.296837 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:48.296886 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:48.296888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.297612 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.297613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:48.298257 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:48.298259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.298653 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.298655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:48.298704 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:48.299232 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:48.299237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:48.301168 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":145,"threads_started":1}
W20260620 08:14:48.301405 2275 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:48.296317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:44645 (local address 127.24.249.190:35415)
0620 08:14:48.296399 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:48.296401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:48.296537 (+ 136us) server_negotiation.cc:407] Connection header received
0620 08:14:48.296574 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:48.296575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:48.296588 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:48.296644 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:48.296956 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.297506 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.298368 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:48.298559 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:48.298622 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:48.301322 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:44645: BlockingRecv error: recv got EOF from 127.24.249.131:44645 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 08:14:49 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:49 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:49.036859 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:49.031308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:53165)
0620 08:14:49.031485 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:49.031497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:49.031560 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:49.031703 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:49.031704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:49.031908 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:49.031969 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:49.031971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.032985 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.032986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:49.033524 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:49.033525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.034057 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.034058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:49.034111 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:49.034610 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:49.034616 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:49.036800 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
W20260620 08:14:49.037045 2279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:49.031440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:53165 (local address 127.24.249.190:35415)
0620 08:14:49.031577 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:49.031579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:49.031588 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:49.031613 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:49.031614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:49.031624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:49.031685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:49.032088 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.032909 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.033635 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.033963 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.034021 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:49.036955 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:53165: BlockingRecv error: recv got EOF from 127.24.249.129:53165 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":52,"threads_started":1}
Jun 20 08:14:49 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:49 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:49.162586 2280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:49.157578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:33385)
0620 08:14:49.157815 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:49.157829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:49.157943 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:49.158105 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:49.158106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:49.158306 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:49.158352 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:49.158355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.159043 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.159044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:49.159594 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:49.159596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.160007 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.160007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:49.160065 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:49.160580 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:49.160585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:49.162529 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
W20260620 08:14:49.162712 2279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:49.157728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:33385 (local address 127.24.249.190:35415)
0620 08:14:49.157830 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:49.157832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:49.157909 (+ 77us) server_negotiation.cc:407] Connection header received
0620 08:14:49.158027 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:49.158028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:49.158042 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:49.158090 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:49.158437 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.158950 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.159707 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.159919 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.160005 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:49.162633 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:33385: BlockingRecv error: recv got EOF from 127.24.249.130:33385 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 08:14:49 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:49 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:49.306449 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:49.301692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:49665)
0620 08:14:49.301915 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:49.301928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:49.302017 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:49.302164 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:49.302165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:49.302347 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:49.302390 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:49.302392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.303120 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.303122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:49.303679 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:49.303681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.304048 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.304049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:49.304117 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:49.304573 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:49.304577 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:49.306355 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":132,"threads_started":1}
W20260620 08:14:49.306491 2279 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:49.301820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:49665 (local address 127.24.249.190:35415)
0620 08:14:49.301902 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:49.301904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:49.301999 (+ 95us) server_negotiation.cc:407] Connection header received
0620 08:14:49.302073 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:49.302074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:49.302089 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:49.302143 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:49.302473 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.303010 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.303775 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:49.303968 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:49.304048 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:49.306428 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:49665: BlockingRecv error: recv got EOF from 127.24.249.131:49665 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 08:14:50 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:50 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:50.042678 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:50.037400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:44553)
0620 08:14:50.037684 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:50.037695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:50.037830 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:50.038032 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:50.038033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:50.038202 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:50.038254 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:50.038256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.039037 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.039038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:50.039684 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:50.039686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.040060 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.040061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:50.040116 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:50.040660 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:50.040665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:50.042580 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":64,"threads_started":1}
W20260620 08:14:50.042800 2283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:50.037480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:44553 (local address 127.24.249.190:35415)
0620 08:14:50.037592 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:50.037594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:50.037825 (+ 231us) server_negotiation.cc:407] Connection header received
0620 08:14:50.037925 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:50.037925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:50.037936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:50.037995 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:50.038354 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.038952 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.039780 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.039976 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.040035 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:50.042693 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:44553: BlockingRecv error: recv got EOF from 127.24.249.129:44553 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
Jun 20 08:14:50 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:50 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:50.168380 2284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:50.163140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:38221)
0620 08:14:50.163381 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:50.163394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:50.163499 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:50.163770 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:50.163771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:50.163948 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:50.163996 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:50.163998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.164773 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.164774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:50.165371 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:50.165372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.165808 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.165809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:50.165895 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:50.166435 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:50.166446 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:50.168321 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":127,"threads_started":1}
W20260620 08:14:50.168514 2283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:50.163261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:38221 (local address 127.24.249.190:35415)
0620 08:14:50.163379 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:50.163381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:50.163476 (+ 95us) server_negotiation.cc:407] Connection header received
0620 08:14:50.163610 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:50.163611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:50.163624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:50.163701 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:50.164105 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.164677 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.165482 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.165710 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.165806 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:50.168431 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:38221: BlockingRecv error: recv got EOF from 127.24.249.130:38221 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 08:14:50 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:50 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:50.312053 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:50.306994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:52777)
0620 08:14:50.307173 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:50.307187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:50.307282 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:50.307428 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:50.307429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:50.307609 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:50.307657 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:50.307659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.308425 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.308426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:50.308962 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:50.308964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.309358 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.309359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:50.309420 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:50.309962 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:50.309967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:50.311987 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
W20260620 08:14:50.312222 2283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:50.307098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:52777 (local address 127.24.249.190:35415)
0620 08:14:50.307180 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:50.307183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:50.307263 (+ 80us) server_negotiation.cc:407] Connection header received
0620 08:14:50.307351 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:50.307352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:50.307366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:50.307413 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:50.307789 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.308323 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.309060 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:50.309268 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:50.309356 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:50.312142 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:52777: BlockingRecv error: recv got EOF from 127.24.249.131:52777 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 08:14:51 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:51 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:51.054618 2286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:51.043301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:48303)
0620 08:14:51.043460 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:51.043474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:51.043535 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:51.043884 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:51.043885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:51.044082 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:51.044163 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:51.044164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.044972 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.044973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:51.045569 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:51.045571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.046038 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.046039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:51.046093 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:51.052014 (+ 5921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:51.052020 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:51.054549 (+ 2529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260620 08:14:51.054872 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:51.043470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:48303 (local address 127.24.249.190:35415)
0620 08:14:51.043640 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:51.043642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:51.043648 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:14:51.043666 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:51.043667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:51.043676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:51.043761 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:51.044309 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.044845 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.045665 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.045913 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.046030 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:51.054801 (+ 8771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:48303: BlockingRecv error: recv got EOF from 127.24.249.129:48303 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":52,"threads_started":1}
Jun 20 08:14:51 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:51 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:51.174256 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:51.168991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:46921)
0620 08:14:51.169132 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:51.169144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:51.169196 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:51.169340 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:51.169341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:51.169503 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:51.169548 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:51.169550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.170474 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.170476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:51.171044 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:51.171046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.171467 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.171468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:51.171519 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:51.172103 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:51.172107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:51.174187 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260620 08:14:51.174439 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:51.169067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:46921 (local address 127.24.249.190:35415)
0620 08:14:51.169169 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:51.169176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:51.169207 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:14:51.169259 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:51.169259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:51.169271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:51.169320 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:51.169633 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.170374 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.171156 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.171369 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.171440 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:51.174353 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:46921: BlockingRecv error: recv got EOF from 127.24.249.130:46921 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 08:14:51 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:51 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:51.317013 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:51.312528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:46607)
0620 08:14:51.312669 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:51.312682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:51.312742 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:51.312882 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:51.312883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:51.313041 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:51.313086 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:51.313087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.313756 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.313757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:51.314315 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:51.314316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.314656 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.314657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:51.314698 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:51.315194 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:51.315200 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:51.316954 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":125,"threads_started":1}
W20260620 08:14:51.317163 2287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:51.312647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:46607 (local address 127.24.249.190:35415)
0620 08:14:51.312714 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:51.312716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:51.312731 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:14:51.312793 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:51.312794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:51.312806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:51.312862 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:51.313158 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.313667 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.314386 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:51.314576 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:51.314648 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:51.317081 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:46607: BlockingRecv error: recv got EOF from 127.24.249.131:46607 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 08:14:52 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:52 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:52.060524 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:52.055388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:43069)
0620 08:14:52.055535 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:52.055585 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:52.055642 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:52.055876 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:52.055877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:52.056051 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:52.056098 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:52.056100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.056850 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.056851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:52.057435 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:52.057437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.057972 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.057974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:52.058026 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:52.058590 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:52.058602 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:52.060464 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260620 08:14:52.060724 2291 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:52.055489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:43069 (local address 127.24.249.190:35415)
0620 08:14:52.055646 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:52.055648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:52.055686 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:14:52.055708 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:52.055708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:52.055720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:52.055787 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:52.056236 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.056755 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.057540 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.057773 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.057850 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:52.060596 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:43069: BlockingRecv error: recv got EOF from 127.24.249.129:43069 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
Jun 20 08:14:52 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:52 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 08:14:52.179801 2291 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:52.174978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:35427 (local address 127.24.249.190:35415)
0620 08:14:52.175051 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:52.175057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:52.175066 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:52.175095 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:52.175096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:52.175106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:52.175160 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:52.175521 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.176069 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.176824 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.177016 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.177073 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:52.179743 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:35427: BlockingRecv error: recv got EOF from 127.24.249.130:35427 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:14:52.179802 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:52.174765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:35427)
0620 08:14:52.174953 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:52.174964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:52.175022 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:52.175189 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:52.175190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:52.175352 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:52.175407 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:52.175409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.176170 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.176171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:52.176722 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:52.176724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.177084 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.177085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:52.177124 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:52.177604 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:52.177609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:52.179654 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
Jun 20 08:14:52 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:52 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:52.323271 2293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:52.317634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:38375)
0620 08:14:52.317847 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:52.317897 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:52.317953 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:52.318114 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:52.318115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:52.318285 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:52.318332 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:52.318333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.319475 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.319476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:52.320032 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:52.320035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.320530 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.320531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:52.320582 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:52.321229 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:52.321239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:52.323212 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
W20260620 08:14:52.323406 2291 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:52.317719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:38375 (local address 127.24.249.190:35415)
0620 08:14:52.317926 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:52.317928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:52.317940 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:14:52.318007 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:52.318008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:52.318020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:52.318079 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:52.318453 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.319373 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.320234 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:52.320427 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:52.320493 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:52.323317 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:38375: BlockingRecv error: recv got EOF from 127.24.249.131:38375 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Jun 20 08:14:53 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:53 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:53.066270 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:53.061094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:42521)
0620 08:14:53.061505 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:53.061519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:53.061595 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:53.061787 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:53.061788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:53.061981 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:53.062031 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:53.062033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.062722 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.062723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:53.063260 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:53.063262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.063638 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.063639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:53.063692 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:53.064187 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:53.064193 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:53.066177 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":290,"threads_started":1}
W20260620 08:14:53.066347 2295 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:53.061283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:42521 (local address 127.24.249.190:35415)
0620 08:14:53.061537 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:53.061539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:53.061576 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:14:53.061661 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:53.061662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:53.061673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:53.061767 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:53.062109 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.062621 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.063346 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.063552 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.063607 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:53.066271 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:42521: BlockingRecv error: recv got EOF from 127.24.249.129:42521 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":208,"threads_started":1}
Jun 20 08:14:53 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:53 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:53.185596 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:53.180332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:56319)
0620 08:14:53.180628 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:53.180640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:53.180737 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:53.180872 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:53.180873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:53.181060 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:53.181129 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:53.181131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.181831 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.181832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:53.182470 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:53.182472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.182882 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.182884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:53.182942 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:53.183444 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:53.183449 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:53.185535 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":87,"threads_started":1}
W20260620 08:14:53.185813 2295 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:53.180440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:56319 (local address 127.24.249.190:35415)
0620 08:14:53.180625 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:53.180627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:53.180716 (+ 89us) server_negotiation.cc:407] Connection header received
0620 08:14:53.180805 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:53.180806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:53.180819 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:53.180867 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:53.181214 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.181738 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.182557 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.182790 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.182879 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:53.185741 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:56319: BlockingRecv error: recv got EOF from 127.24.249.130:56319 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Jun 20 08:14:53 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:53 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:53.328384 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:53.323756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:49931)
0620 08:14:53.323870 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:53.323883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:53.323937 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:53.324089 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:53.324089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:53.324251 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:53.324296 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:53.324298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.325052 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.325053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:53.325581 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:53.325583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.325962 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.325963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:53.326004 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:53.326434 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:53.326438 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:53.328326 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
W20260620 08:14:53.328557 2295 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:53.323913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:49931 (local address 127.24.249.190:35415)
0620 08:14:53.323958 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:53.323960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:53.323969 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:53.323994 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:53.323994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:53.324009 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:53.324071 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:53.324397 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.324963 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.325657 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:53.325847 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:53.325961 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:53.328449 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:49931: BlockingRecv error: recv got EOF from 127.24.249.131:49931 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 08:14:54 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:54 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:54.072671 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:54.067073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:55953)
0620 08:14:54.067248 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:54.067264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:54.067325 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:54.067627 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:54.067629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:54.067921 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:54.067972 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:54.067975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.068777 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.068778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:54.069343 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:54.069344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.069843 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.069844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:54.069933 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:54.070469 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:54.070475 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:54.072596 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":122,"threads_started":1}
W20260620 08:14:54.072831 2299 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:54.067195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:55953 (local address 127.24.249.190:35415)
0620 08:14:54.067418 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:54.067420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:54.067431 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:54.067475 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:54.067476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:54.067487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:54.067563 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:54.068096 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.068652 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.069438 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.069725 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.069808 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:54.072719 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:55953: BlockingRecv error: recv got EOF from 127.24.249.129:55953 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":163,"threads_started":1}
Jun 20 08:14:54 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:54 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:54.191845 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:54.186182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:39987)
0620 08:14:54.186403 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:54.186418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:54.186495 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:54.186686 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:54.186687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:54.186913 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:54.186959 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:54.186961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.187730 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.187731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:54.188348 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:54.188350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.188798 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.188799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:54.188862 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:54.189359 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:54.189364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:54.191786 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":114,"threads_started":1}
W20260620 08:14:54.192032 2299 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:54.186328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:39987 (local address 127.24.249.190:35415)
0620 08:14:54.186408 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:54.186411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:54.186473 (+ 62us) server_negotiation.cc:407] Connection header received
0620 08:14:54.186569 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:54.186569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:54.186584 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:54.186665 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:54.187051 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.187608 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.188446 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.188672 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.188798 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:54.191921 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:39987: BlockingRecv error: recv got EOF from 127.24.249.130:39987 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 08:14:54 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:54 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:54.333801 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:54.328953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:58477)
0620 08:14:54.329132 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:54.329142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:54.329189 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:54.329340 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:54.329341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:54.329503 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:54.329557 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:54.329558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.330254 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.330255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:54.330796 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:54.330797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.331112 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.331113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:54.331151 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:54.331593 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:54.331598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:54.333708 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
W20260620 08:14:54.333861 2299 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:54.329032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:58477 (local address 127.24.249.190:35415)
0620 08:14:54.329204 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:54.329206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:54.329245 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:14:54.329268 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:54.329269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:54.329281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:54.329336 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:54.329626 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.330146 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.330873 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:54.331046 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:54.331097 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:54.333792 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:58477: BlockingRecv error: recv got EOF from 127.24.249.131:58477 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Jun 20 08:14:55 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:55 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:55.078284 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:55.073305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:37561)
0620 08:14:55.073456 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:55.073468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:55.073521 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:55.073702 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:55.073703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:55.073940 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:55.073987 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:55.073989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.074889 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.074890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:55.075425 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:55.075427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.075813 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.075814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:55.075860 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:55.076334 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:55.076338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:55.078194 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
W20260620 08:14:55.078382 2303 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:55.073387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:37561 (local address 127.24.249.190:35415)
0620 08:14:55.073535 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:55.073538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:55.073549 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:55.073578 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:55.073579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:55.073593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:55.073678 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:55.074066 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.074774 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.075531 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.075750 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.075813 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:55.078303 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:37561: BlockingRecv error: recv got EOF from 127.24.249.129:37561 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
Jun 20 08:14:55 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:55 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:55.197587 2304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:55.192413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:38977)
0620 08:14:55.192667 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:55.192678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:55.192776 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:55.192924 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:55.192925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:55.193108 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:55.193171 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:55.193173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.193929 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.193932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:55.194464 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:55.194466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.194873 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.194874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:55.194940 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:55.195428 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:55.195433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:55.197490 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
W20260620 08:14:55.197677 2303 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:55.192503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:38977 (local address 127.24.249.190:35415)
0620 08:14:55.192597 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:55.192600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:55.192762 (+ 162us) server_negotiation.cc:407] Connection header received
0620 08:14:55.192847 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:55.192848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:55.192862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:55.192908 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:55.193266 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.193797 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.194565 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.194780 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.194871 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:55.197573 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:38977: BlockingRecv error: recv got EOF from 127.24.249.130:38977 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 08:14:55 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:55 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:55.339717 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:55.334456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:52977)
0620 08:14:55.334594 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:55.334605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:55.334717 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:55.334870 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:55.334870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:55.335044 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:55.335089 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:55.335090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.335834 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.335836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:55.336435 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:55.336437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.336864 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.336864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:55.336915 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:55.337484 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:55.337490 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:55.339627 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260620 08:14:55.339803 2303 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:55.334485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:52977 (local address 127.24.249.190:35415)
0620 08:14:55.334560 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:55.334562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:55.334717 (+ 155us) server_negotiation.cc:407] Connection header received
0620 08:14:55.334782 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:55.334783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:55.334799 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:55.334859 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:55.335156 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.335722 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.336533 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:55.336747 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:55.336827 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:55.339740 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:52977: BlockingRecv error: recv got EOF from 127.24.249.131:52977 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 08:14:56 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:56 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:56.088806 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:56.078907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:36699)
0620 08:14:56.079107 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:56.079119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:56.079214 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:56.079497 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:56.079498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:56.079685 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:56.079737 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:56.079739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.080446 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.080448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:56.080973 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:56.080974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.081359 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.081360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:56.081406 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:56.086706 (+ 5300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:56.086722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:56.088742 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":109,"threads_started":1}
W20260620 08:14:56.089040 2307 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:56.079037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:36699 (local address 127.24.249.190:35415)
0620 08:14:56.079149 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:56.079151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:56.079164 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:14:56.079356 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:56.079358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:56.079370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:56.079438 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:56.079819 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.080349 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.081057 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.081253 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.081335 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:56.088975 (+ 7640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:36699: BlockingRecv error: recv got EOF from 127.24.249.129:36699 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
Jun 20 08:14:56 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:56 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:56.202863 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:56.198078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:42129)
0620 08:14:56.198217 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:56.198231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:56.198305 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:56.198438 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:56.198439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:56.198600 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:56.198643 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:56.198645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.199331 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.199332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:56.199949 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:56.199952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.200371 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.200372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:56.200419 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:56.200908 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:56.200914 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:56.202803 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
W20260620 08:14:56.203013 2307 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:56.198241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:42129 (local address 127.24.249.190:35415)
0620 08:14:56.198303 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:56.198310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:56.198337 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:14:56.198362 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:56.198363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:56.198373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:56.198413 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:56.198709 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.199219 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.200057 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.200281 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.200346 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:56.202916 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:42129: BlockingRecv error: recv got EOF from 127.24.249.130:42129 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 08:14:56 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:56 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:56.345103 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:56.340313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:42137)
0620 08:14:56.340451 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:56.340466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:56.340515 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:56.340643 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:56.340643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:56.340841 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:56.340886 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:56.340888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.341530 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.341531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:56.342136 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:56.342138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.342494 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.342495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:56.342537 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:56.343029 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:56.343034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:56.344988 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260620 08:14:56.345142 2307 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:56.340357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:42137 (local address 127.24.249.190:35415)
0620 08:14:56.340526 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:56.340527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:56.340536 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:56.340556 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:56.340557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:56.340568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:56.340623 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:56.340960 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.341449 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.342224 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:56.342417 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:56.342484 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:56.345077 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:42137: BlockingRecv error: recv got EOF from 127.24.249.131:42137 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Jun 20 08:14:57 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:57 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:57.094280 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:57.089324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:34813)
0620 08:14:57.089464 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:57.089475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:57.089525 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:57.089767 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:57.089768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:57.089960 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:57.090007 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:57.090009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.090743 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.090744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:57.091296 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:57.091298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.091730 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.091731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:57.091780 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:57.092355 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:57.092359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:57.094210 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260620 08:14:57.094475 2311 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:57.089449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:34813 (local address 127.24.249.190:35415)
0620 08:14:57.089577 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:57.089597 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:57.089606 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:14:57.089635 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:57.089635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:57.089648 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:57.089724 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:57.090091 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.090638 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.091396 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.091596 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.091680 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:57.094389 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:34813: BlockingRecv error: recv got EOF from 127.24.249.129:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
Jun 20 08:14:57 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:57 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:57.209460 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:57.203507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:37743)
0620 08:14:57.203706 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:57.203720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:57.203803 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:57.204023 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:57.204024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:57.204471 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:57.204521 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:57.204524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.205324 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.205325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:57.205913 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:57.205915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.206390 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.206392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:57.206443 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:57.207047 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:57.207052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:57.209395 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":145,"threads_started":1}
W20260620 08:14:57.209599 2311 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:57.203673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:37743 (local address 127.24.249.190:35415)
0620 08:14:57.203815 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:57.203853 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:57.203863 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:57.203891 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:57.203892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:57.203902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:57.203961 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:57.204628 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.205207 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.206026 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.206222 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.206366 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:57.209512 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:37743: BlockingRecv error: recv got EOF from 127.24.249.130:37743 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 20 08:14:57 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:57 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.131@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:57.351344 2313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:57.345695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.131:35773)
0620 08:14:57.346025 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:57.346037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:57.346188 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:57.346362 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:57.346363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:57.346529 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:57.346592 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:57.346593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.347333 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.347334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:57.347947 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:57.347948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.348369 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.348370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:57.348476 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:57.349040 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:57.349045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:57.351240 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":97,"threads_started":1}
W20260620 08:14:57.351562 2311 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:57.345977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:35773 (local address 127.24.249.190:35415)
0620 08:14:57.346115 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:57.346117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:57.346184 (+ 67us) server_negotiation.cc:407] Connection header received
0620 08:14:57.346255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:57.346255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:57.346268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:57.346350 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:57.346694 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.347228 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.348060 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:57.348280 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:57.348346 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:57.351366 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.131:35773: BlockingRecv error: recv got EOF from 127.24.249.131:35773 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 08:14:58 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:58 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.129@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:58.099992 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.094892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.129:53463)
0620 08:14:58.095049 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:58.095060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:58.095121 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:58.095370 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:58.095372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:58.095545 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:58.095592 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.095594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.096402 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.096403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.096949 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.096950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.097380 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.097382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.097465 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.097976 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:58.097981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:58.099889 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
W20260620 08:14:58.100088 2315 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:58.095030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:53463 (local address 127.24.249.190:35415)
0620 08:14:58.095167 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:58.095169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:58.095179 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:14:58.095220 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:58.095220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:58.095232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:58.095295 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:58.095695 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.096297 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.097043 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.097285 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.097371 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.099978 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.129:53463: BlockingRecv error: recv got EOF from 127.24.249.129:53463 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
Jun 20 08:14:58 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 08:14:58 dist-test-slave-65w8 krb5kdc[1257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.249.130@KRBTEST.COM for oryx/127.24.249.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 08:14:58.215665 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.210152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:35415 (local address 127.24.249.130:34755)
0620 08:14:58.210324 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:58.210343 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:58.210403 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:58.210626 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:58.210628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:58.210808 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:58.210858 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.210860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.211842 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.211843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.212603 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.212606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.213040 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.213041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.213085 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.213681 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:14:58.213685 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:58.215593 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.249.190:35415: Server oryx/127.24.249.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
W20260620 08:14:58.215878 2315 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:58.210370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:34755 (local address 127.24.249.190:35415)
0620 08:14:58.210450 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:58.210452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:58.210463 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:14:58.210503 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:58.210504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:58.210522 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:58.210579 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:58.210967 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.211656 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.212762 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.212956 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.213019 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.215806 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.249.130:34755: BlockingRecv error: recv got EOF from 127.24.249.130:34755 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 08:14:58.267163 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 1840
I20260620 08:14:58.272251 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 1992
I20260620 08:14:58.277000 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 2125
I20260620 08:14:58.281185 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 1770
2026-06-20T08:14:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17567 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T08:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:58Z Disabled control of system clock
I20260620 08:14:58.310616 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:36795
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 08:14:58.394946 2320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:58.395247 2320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:58.395270 2320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:58.396449 2320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:58.396487 2320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:58.396499 2320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:58.396512 2320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:58.396524 2320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:58.397833 2320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:36795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:58.398106 2320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:58.398361 2320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:58.405644 2326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.405773 2328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.405916 2325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:58.406924 2320 server_base.cc:1034] running on GCE node
I20260620 08:14:58.407310 2320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:58.407721 2320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:58.408915 2320 hybrid_clock.cc:648] HybridClock initialized: now 1781943298408896 us; error 39 us; skew 500 ppm
I20260620 08:14:58.410351 2320 webserver.cc:460] Webserver started at http://127.24.249.190:43729/ using document root <none> and password file <none>
I20260620 08:14:58.410554 2320 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:58.410595 2320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:58.410687 2320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:58.411803 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "1ec4aae64ae149d08a7a75ea9530b479"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "ed9df1b22ec1c4de411b2684430e8347"
server_key_iv: "306dc7f4bd0de0fd9363a8b5ba4b3c14"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.412106 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "1ec4aae64ae149d08a7a75ea9530b479"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "ed9df1b22ec1c4de411b2684430e8347"
server_key_iv: "306dc7f4bd0de0fd9363a8b5ba4b3c14"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.413395 2320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:58.414412 2334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.414661 2320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:58.414753 2320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "1ec4aae64ae149d08a7a75ea9530b479"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "ed9df1b22ec1c4de411b2684430e8347"
server_key_iv: "306dc7f4bd0de0fd9363a8b5ba4b3c14"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.414845 2320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:58.423990 2320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:58.424299 2320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:58.424407 2320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:58.428304 2320 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:36795
I20260620 08:14:58.428354 2386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:36795 every 8 connection(s)
I20260620 08:14:58.428685 2320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:58.429378 2387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:58.432065 2387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479: Bootstrap starting.
I20260620 08:14:58.432651 2387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:58.432940 2387 log.cc:824] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:58.433646 2387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479: No bootstrap required, opened a new log
I20260620 08:14:58.435290 2387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4aae64ae149d08a7a75ea9530b479" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36795 } }
I20260620 08:14:58.435461 2387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:58.435510 2387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec4aae64ae149d08a7a75ea9530b479, State: Initialized, Role: FOLLOWER
I20260620 08:14:58.435714 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 2320
I20260620 08:14:58.435738 2387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4aae64ae149d08a7a75ea9530b479" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36795 } }
I20260620 08:14:58.435817 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:58.435840 2387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:58.435874 2387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:58.435937 2387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:58.436028 25574 external_mini_cluster.cc:1428] Setting key c7b7db9804ebeef46b310cae6924a96d
I20260620 08:14:58.436686 2387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4aae64ae149d08a7a75ea9530b479" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36795 } }
I20260620 08:14:58.436811 2387 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec4aae64ae149d08a7a75ea9530b479; no voters:
I20260620 08:14:58.436995 2387 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:58.437037 2392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:58.437223 2392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [term 1 LEADER]: Becoming Leader. State: Replica: 1ec4aae64ae149d08a7a75ea9530b479, State: Running, Role: LEADER
I20260620 08:14:58.437258 2387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:58.437417 2392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4aae64ae149d08a7a75ea9530b479" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36795 } }
I20260620 08:14:58.438005 2393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec4aae64ae149d08a7a75ea9530b479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4aae64ae149d08a7a75ea9530b479" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36795 } } }
I20260620 08:14:58.438094 2393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:58.438320 2401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:58.438848 2401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:58.439205 2394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec4aae64ae149d08a7a75ea9530b479. Latest consensus state: current_term: 1 leader_uuid: "1ec4aae64ae149d08a7a75ea9530b479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec4aae64ae149d08a7a75ea9530b479" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 36795 } } }
I20260620 08:14:58.439379 2394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479 [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:58.440478 2401 catalog_manager.cc:1347] Generated new cluster ID: ce45171d43af4987a81d30a28bf28f79
I20260620 08:14:58.440528 2401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:58.442672 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.436578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.24.249.190:36795)
0620 08:14:58.436834 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:58.436845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:58.436900 (+ 55us) server_negotiation.cc:407] Connection header received
0620 08:14:58.437040 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:58.437048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:58.437124 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:58.437599 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:58.437710 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.439243 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.440313 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.440905 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.441013 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.441831 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:58.441835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:58.441836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:58.442066 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:58.442106 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:58.442120 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:58.442184 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:58.442301 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:14:58.442458 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260620 08:14:58.447419 2401 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:58.447965 2401 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:58.452883 2401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec4aae64ae149d08a7a75ea9530b479: Generated new TSK 0
I20260620 08:14:58.453071 2401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:14:58.455775 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:36795
--builtin_ntp_servers=127.24.249.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 08:14:58.542809 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:58.543076 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:58.543114 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:58.544377 2411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:58.544452 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:58.544498 2411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:58.546017 2411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:58.546293 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:58.546618 2411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:58.552096 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.552068 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.552031 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:58.552436 2411 server_base.cc:1034] running on GCE node
I20260620 08:14:58.552641 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:58.552919 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:58.554086 2411 hybrid_clock.cc:648] HybridClock initialized: now 1781943298554070 us; error 31 us; skew 500 ppm
I20260620 08:14:58.555419 2411 webserver.cc:460] Webserver started at http://127.24.249.129:39065/ using document root <none> and password file <none>
I20260620 08:14:58.555655 2411 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:58.555727 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:58.555872 2411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:58.557029 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "1028904f1a8d42d588c0deb9099d0b6f"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "fe1eecc2e69c32a21107379c901e75be"
server_key_iv: "f6463c5b90518022a85cb1df1b87e581"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.557391 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "1028904f1a8d42d588c0deb9099d0b6f"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "fe1eecc2e69c32a21107379c901e75be"
server_key_iv: "f6463c5b90518022a85cb1df1b87e581"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.558858 2411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:14:58.559729 2425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.559950 2411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:58.560042 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "1028904f1a8d42d588c0deb9099d0b6f"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "fe1eecc2e69c32a21107379c901e75be"
server_key_iv: "f6463c5b90518022a85cb1df1b87e581"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.560111 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:58.582240 2411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:58.582523 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:58.582633 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:58.582819 2411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:58.583175 2411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:58.583216 2411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.583242 2411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:58.583292 2411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.588996 2411 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:38803
I20260620 08:14:58.589099 2538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:38803 every 8 connection(s)
I20260620 08:14:58.589432 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:58.591379 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 2411
I20260620 08:14:58.591487 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:58.591670 25574 external_mini_cluster.cc:1428] Setting key d434c6e8ccb618883b2d1db6ba345f94
I20260620 08:14:58.592739 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:36795
--builtin_ntp_servers=127.24.249.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 08:14:58.595980 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.590240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:43999 (local address 127.24.249.190:36795)
0620 08:14:58.590339 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:58.590342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:58.590682 (+ 340us) server_negotiation.cc:407] Connection header received
0620 08:14:58.591308 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:58.591309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:58.591324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:58.591391 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:58.592371 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.593166 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.594357 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.594558 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.594618 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.595708 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:58.595710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:58.595710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:58.595718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:58.595733 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:58.595737 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:58.595788 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:58.595886 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 08:14:58.595922 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 08:14:58.596264 2541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.590200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:36795 (local address 127.24.249.129:43999)
0620 08:14:58.590501 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:58.590539 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:58.591168 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:58.591473 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:58.591476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:58.591504 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:58.592222 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.592230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.593283 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.593284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.594260 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.594263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.594694 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.594696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.594905 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.595538 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:58.595539 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:58.595579 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:58.595610 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:58.595804 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:58.595809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:58.595812 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:58.595884 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:14:58.596004 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260620 08:14:58.596889 2539 heartbeater.cc:347] Connected to a master server at 127.24.249.190:36795
I20260620 08:14:58.596984 2539 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:58.597184 2539 heartbeater.cc:511] Master 127.24.249.190:36795 requested a full tablet report, sending...
I20260620 08:14:58.597800 2351 ts_manager.cc:194] Registered new tserver with Master: 1028904f1a8d42d588c0deb9099d0b6f (127.24.249.129:38803)
I20260620 08:14:58.598455 2351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.129:43999
W20260620 08:14:58.675525 2542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:58.675765 2542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:58.675800 2542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:58.676987 2542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:58.677053 2542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:58.677096 2542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:58.678578 2542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:58.678845 2542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:58.679152 2542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:58.684445 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.684510 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.684545 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:58.684741 2542 server_base.cc:1034] running on GCE node
I20260620 08:14:58.684939 2542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:58.685177 2542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:58.686363 2542 hybrid_clock.cc:648] HybridClock initialized: now 1781943298686346 us; error 25 us; skew 500 ppm
I20260620 08:14:58.687619 2542 webserver.cc:460] Webserver started at http://127.24.249.130:35759/ using document root <none> and password file <none>
I20260620 08:14:58.687892 2542 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:58.687966 2542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:58.688117 2542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:58.689221 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "6da54d2095234220ae22e738dda82086"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "c9ba4ec7339b804bd418a98cd9cf776d"
server_key_iv: "ed0b014216719da31eb82f56ae2ea314"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.689582 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "6da54d2095234220ae22e738dda82086"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "c9ba4ec7339b804bd418a98cd9cf776d"
server_key_iv: "ed0b014216719da31eb82f56ae2ea314"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.691005 2542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:58.691813 2556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.692008 2542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:58.692106 2542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "6da54d2095234220ae22e738dda82086"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "c9ba4ec7339b804bd418a98cd9cf776d"
server_key_iv: "ed0b014216719da31eb82f56ae2ea314"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.692194 2542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:58.708371 2542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:58.708710 2542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:58.708870 2542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:58.709110 2542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:58.709434 2542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:58.709491 2542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.709535 2542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:58.709558 2542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.715216 2542 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:44495
I20260620 08:14:58.715322 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:44495 every 8 connection(s)
I20260620 08:14:58.715626 2542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:58.718596 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 2542
I20260620 08:14:58.718695 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:58.718888 25574 external_mini_cluster.cc:1428] Setting key e39064ed19b1aa61fe3283a6f3e55d47
I20260620 08:14:58.719779 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:36795
--builtin_ntp_servers=127.24.249.148:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 08:14:58.721427 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.716387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:52787 (local address 127.24.249.190:36795)
0620 08:14:58.716454 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:58.716456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:58.716682 (+ 226us) server_negotiation.cc:407] Connection header received
0620 08:14:58.717162 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:58.717163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:58.717174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:58.717245 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:58.717961 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.718599 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.719866 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.720164 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.720243 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.721091 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:58.721093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:58.721094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:58.721099 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:58.721118 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:58.721123 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:58.721185 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:58.721302 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:14:58.721337 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:14:58.721725 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.716376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:36795 (local address 127.24.249.130:52787)
0620 08:14:58.716549 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:58.716589 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:58.717072 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:58.717300 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:58.717303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:58.717311 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:58.717821 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.717825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.718697 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.718698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.719768 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.719770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.720247 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.720248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.720372 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.720948 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:58.720948 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:58.720976 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:58.720999 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:58.721197 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:58.721202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:58.721206 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:14:58.721284 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 08:14:58.721408 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 08:14:58.722225 2670 heartbeater.cc:347] Connected to a master server at 127.24.249.190:36795
I20260620 08:14:58.722306 2670 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:58.722494 2670 heartbeater.cc:511] Master 127.24.249.190:36795 requested a full tablet report, sending...
I20260620 08:14:58.722893 2351 ts_manager.cc:194] Registered new tserver with Master: 6da54d2095234220ae22e738dda82086 (127.24.249.130:44495)
I20260620 08:14:58.723282 2351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.130:52787
W20260620 08:14:58.802214 2673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:58.802448 2673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:58.802484 2673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:58.803658 2673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:58.803722 2673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:58.803766 2673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:58.805181 2673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:58.805438 2673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:58.805725 2673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:58.810810 2679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.810806 2678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.810824 2681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:58.810860 2673 server_base.cc:1034] running on GCE node
I20260620 08:14:58.811333 2673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:58.811559 2673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:58.812700 2673 hybrid_clock.cc:648] HybridClock initialized: now 1781943298812675 us; error 52 us; skew 500 ppm
I20260620 08:14:58.814112 2673 webserver.cc:460] Webserver started at http://127.24.249.131:40343/ using document root <none> and password file <none>
I20260620 08:14:58.814357 2673 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:58.814405 2673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:58.814558 2673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:58.815639 2673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "b8a82c42fe4e41abad4d985ba214cae3"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "ec192415f28487fdb849aa7735fa676e"
server_key_iv: "cebd896d823702bffc757fd676979117"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.816010 2673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "b8a82c42fe4e41abad4d985ba214cae3"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "ec192415f28487fdb849aa7735fa676e"
server_key_iv: "cebd896d823702bffc757fd676979117"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.817391 2673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:58.818295 2687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.818547 2673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:14:58.818671 2673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "b8a82c42fe4e41abad4d985ba214cae3"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "ec192415f28487fdb849aa7735fa676e"
server_key_iv: "cebd896d823702bffc757fd676979117"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.818749 2673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:58.839838 2673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:58.840193 2673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:58.840333 2673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:58.840567 2673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:58.840911 2673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:58.840945 2673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.840967 2673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:58.841010 2673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.846774 2673 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:32817
I20260620 08:14:58.846854 2800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:32817 every 8 connection(s)
I20260620 08:14:58.847172 2673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:58.852382 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.847955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:43803 (local address 127.24.249.190:36795)
0620 08:14:58.848015 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:58.848018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:58.848244 (+ 226us) server_negotiation.cc:407] Connection header received
0620 08:14:58.848694 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:58.848695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:58.848706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:58.848799 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:58.849503 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.850094 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.851088 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.851279 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.851331 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.852101 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:58.852103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:58.852103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:58.852108 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:58.852121 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:58.852125 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:58.852181 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:58.852298 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:14:58.852324 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:14:58.852567 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:58.847882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:36795 (local address 127.24.249.131:43803)
0620 08:14:58.848078 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:58.848118 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:58.848597 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:58.848903 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:58.848907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:58.848916 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:58.849419 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.849424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.850187 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.850188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.851012 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:14:58.851014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:14:58.851354 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:14:58.851355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:14:58.851459 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:14:58.851956 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:58.851956 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:58.851984 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:58.852005 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:58.852209 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:58.852214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:58.852216 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:58.852290 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:14:58.852367 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260620 08:14:58.853153 2801 heartbeater.cc:347] Connected to a master server at 127.24.249.190:36795
I20260620 08:14:58.853240 2801 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:58.853464 2801 heartbeater.cc:511] Master 127.24.249.190:36795 requested a full tablet report, sending...
I20260620 08:14:58.853967 2351 ts_manager.cc:194] Registered new tserver with Master: b8a82c42fe4e41abad4d985ba214cae3 (127.24.249.131:32817)
I20260620 08:14:58.854337 2351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.131:43803
I20260620 08:14:58.855886 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 2673
I20260620 08:14:58.855993 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:58.856178 25574 external_mini_cluster.cc:1428] Setting key c6330e3fd8aeadd79263805d1fd04d44
I20260620 08:14:58.857100 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:14:58.858899 2391 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:58.858413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.24.249.190:36795)
0620 08:14:58.858448 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:58.858450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:58.858569 (+ 119us) server_negotiation.cc:407] Connection header received
0620 08:14:58.858613 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:58.858614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:58.858630 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:58.858681 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:58.858818 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57612: BlockingRecv error: recv got EOF from 127.0.0.1:57612 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:14:58.858929 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 2411
I20260620 08:14:58.863616 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 2542
I20260620 08:14:58.868474 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 2673
I20260620 08:14:58.872874 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 2320
2026-06-20T08:14:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (591 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T08:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:14:58Z Disabled control of system clock
I20260620 08:14:58.901513 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40007
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:40007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 08:14:58.982617 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:58.982873 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:58.982910 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:58.984091 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:14:58.984165 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:58.984189 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:58.984222 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:14:58.984248 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:14:58.985594 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:40007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:58.985913 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:58.986217 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:58.990945 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.990958 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:58.990972 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:58.991269 2812 server_base.cc:1034] running on GCE node
I20260620 08:14:58.991566 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:58.991855 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:58.993031 2812 hybrid_clock.cc:648] HybridClock initialized: now 1781943298993025 us; error 55 us; skew 500 ppm
I20260620 08:14:58.994371 2812 webserver.cc:460] Webserver started at http://127.24.249.190:36211/ using document root <none> and password file <none>
I20260620 08:14:58.994629 2812 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:58.994702 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:58.994858 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:58.996012 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "2f6a04d557d44fb6bf0505dec11789cc"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "52c6a7faeee7e63ff27b940ad1bedab1"
server_key_iv: "16e6fb6f7d654ca65ce4bba7d3513057"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.996373 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "2f6a04d557d44fb6bf0505dec11789cc"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "52c6a7faeee7e63ff27b940ad1bedab1"
server_key_iv: "16e6fb6f7d654ca65ce4bba7d3513057"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.997761 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:14:58.998706 2826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:58.998932 2812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:58.999049 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "2f6a04d557d44fb6bf0505dec11789cc"
format_stamp: "Formatted at 2026-06-20 08:14:58 on dist-test-slave-65w8"
server_key: "52c6a7faeee7e63ff27b940ad1bedab1"
server_key_iv: "16e6fb6f7d654ca65ce4bba7d3513057"
server_key_version: "encryptionkey@0"
I20260620 08:14:58.999145 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:59.003213 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:59.003649 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:59.003847 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:59.008359 2812 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:40007
I20260620 08:14:59.008522 2878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:40007 every 8 connection(s)
I20260620 08:14:59.008821 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:14:59.009528 2879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:14:59.012344 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc: Bootstrap starting.
I20260620 08:14:59.012892 2879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc: Neither blocks nor log segments found. Creating new log.
I20260620 08:14:59.013134 2879 log.cc:824] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc: Log is configured to *not* fsync() on all Append() calls
I20260620 08:14:59.013792 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc: No bootstrap required, opened a new log
I20260620 08:14:59.015427 2879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6a04d557d44fb6bf0505dec11789cc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40007 } }
I20260620 08:14:59.015570 2879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:14:59.015594 2879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6a04d557d44fb6bf0505dec11789cc, State: Initialized, Role: FOLLOWER
I20260620 08:14:59.015738 2879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6a04d557d44fb6bf0505dec11789cc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40007 } }
I20260620 08:14:59.015827 2879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:14:59.015851 2879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:14:59.015880 2879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:14:59.016311 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 2812
I20260620 08:14:59.016423 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:14:59.016441 2879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6a04d557d44fb6bf0505dec11789cc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40007 } }
I20260620 08:14:59.016544 2879 leader_election.cc:302] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f6a04d557d44fb6bf0505dec11789cc; no voters:
I20260620 08:14:59.016657 25574 external_mini_cluster.cc:1428] Setting key 78ec8dd0c4cdcc15d851be20fb94f09b
I20260620 08:14:59.016731 2879 leader_election.cc:288] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:14:59.016762 2882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:14:59.016914 2882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [term 1 LEADER]: Becoming Leader. State: Replica: 2f6a04d557d44fb6bf0505dec11789cc, State: Running, Role: LEADER
I20260620 08:14:59.017019 2882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6a04d557d44fb6bf0505dec11789cc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40007 } }
I20260620 08:14:59.017357 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6a04d557d44fb6bf0505dec11789cc. Latest consensus state: current_term: 1 leader_uuid: "2f6a04d557d44fb6bf0505dec11789cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6a04d557d44fb6bf0505dec11789cc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40007 } } }
I20260620 08:14:59.017373 2879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:14:59.017467 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:59.017678 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:14:59.017396 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f6a04d557d44fb6bf0505dec11789cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6a04d557d44fb6bf0505dec11789cc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40007 } } }
I20260620 08:14:59.018134 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc [sys.catalog]: This master's current role is: LEADER
I20260620 08:14:59.018335 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:14:59.020217 2888 catalog_manager.cc:1347] Generated new cluster ID: 8d338065fe394e86b3989ee7b473c606
I20260620 08:14:59.020296 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:14:59.020378 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:59.017247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.24.249.190:40007)
0620 08:14:59.017700 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:59.017710 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:59.018428 (+ 718us) server_negotiation.cc:407] Connection header received
0620 08:14:59.018551 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:59.018558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:59.018605 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:59.018732 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:59.019854 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:59.019857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:59.019858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:59.019861 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:59.019898 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:59.019909 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:59.020002 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:59.020079 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 08:14:59.020138 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":93,"threads_started":1}
I20260620 08:14:59.027050 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:14:59.027223 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:14:59.032851 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f6a04d557d44fb6bf0505dec11789cc: Generated new TSK 0
I20260620 08:14:59.033056 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:14:59.036736 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40007
--builtin_ntp_servers=127.24.249.148:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 08:14:59.121721 2903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:59.122028 2903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:59.122071 2903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:59.123323 2903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:59.123394 2903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:59.123440 2903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:14:59.124866 2903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:59.125180 2903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:59.125519 2903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:59.131974 2911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:59.131978 2909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:59.131978 2908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:59.132244 2903 server_base.cc:1034] running on GCE node
I20260620 08:14:59.132474 2903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:59.132720 2903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:59.133952 2903 hybrid_clock.cc:648] HybridClock initialized: now 1781943299133914 us; error 47 us; skew 500 ppm
I20260620 08:14:59.135514 2903 webserver.cc:460] Webserver started at http://127.24.249.129:41879/ using document root <none> and password file <none>
I20260620 08:14:59.135826 2903 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:59.136065 2903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:59.136252 2903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:59.137372 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "4c17c57270bc44359b770be76d7dd103"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "baa8364c5ba533268a2fce9f5749af8d"
server_key_iv: "f8cea3f9ba8cc0db8e89d3df4c375134"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.137763 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "4c17c57270bc44359b770be76d7dd103"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "baa8364c5ba533268a2fce9f5749af8d"
server_key_iv: "f8cea3f9ba8cc0db8e89d3df4c375134"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.139217 2903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:14:59.140023 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.140182 2903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:59.140293 2903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "4c17c57270bc44359b770be76d7dd103"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "baa8364c5ba533268a2fce9f5749af8d"
server_key_iv: "f8cea3f9ba8cc0db8e89d3df4c375134"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.140375 2903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:59.143920 2903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:59.144217 2903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:59.144357 2903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:59.144596 2903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:59.145042 2903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:59.145079 2903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.145102 2903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:59.145134 2903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.153988 2903 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:41661
I20260620 08:14:59.154016 3030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:41661 every 8 connection(s)
I20260620 08:14:59.154420 2903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:14:59.154991 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 2903
I20260620 08:14:59.155100 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:14:59.155347 25574 external_mini_cluster.cc:1428] Setting key 90821c66718f190ca005e4b57d6385a7
I20260620 08:14:59.156606 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40007
--builtin_ntp_servers=127.24.249.148:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 08:14:59.159574 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:59.155690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:58719 (local address 127.24.249.190:40007)
0620 08:14:59.155751 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:59.155753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:59.156088 (+ 335us) server_negotiation.cc:407] Connection header received
0620 08:14:59.156822 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:59.156823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:59.156831 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:59.156864 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:59.159285 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:59.159287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:59.159287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:59.159292 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:59.159310 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:59.159314 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:59.159377 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:59.159498 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 08:14:59.159513 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:14:59.159818 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:59.155268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40007 (local address 127.24.249.129:58719)
0620 08:14:59.155688 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:59.155737 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:59.156685 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:59.157075 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:59.157080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:59.157091 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:59.159022 (+ 1931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:59.159022 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:59.159070 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:59.159115 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:59.159401 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:59.159408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:59.159413 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:14:59.159502 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:14:59.159579 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260620 08:14:59.160434 3031 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40007
I20260620 08:14:59.160534 3031 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:59.160694 3031 heartbeater.cc:511] Master 127.24.249.190:40007 requested a full tablet report, sending...
I20260620 08:14:59.161245 2843 ts_manager.cc:194] Registered new tserver with Master: 4c17c57270bc44359b770be76d7dd103 (127.24.249.129:41661)
W20260620 08:14:59.241165 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:59.241422 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:59.241459 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:59.242730 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:59.242800 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:59.242842 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:14:59.244339 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:59.244622 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:59.244936 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:59.249969 3042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:59.250007 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:59.249969 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:59.250244 3034 server_base.cc:1034] running on GCE node
I20260620 08:14:59.250464 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:59.250720 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:59.251919 3034 hybrid_clock.cc:648] HybridClock initialized: now 1781943299251910 us; error 36 us; skew 500 ppm
I20260620 08:14:59.253302 3034 webserver.cc:460] Webserver started at http://127.24.249.130:40065/ using document root <none> and password file <none>
I20260620 08:14:59.253551 3034 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:59.253618 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:59.253754 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:59.254926 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "8a8533fbd1d84474b30cfef24c59bdd1"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "f0d19db1611d7266eac4f4aec1a55c0f"
server_key_iv: "ded6a7b38c0e923817d77918cef1e268"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.255292 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "8a8533fbd1d84474b30cfef24c59bdd1"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "f0d19db1611d7266eac4f4aec1a55c0f"
server_key_iv: "ded6a7b38c0e923817d77918cef1e268"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.256574 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:14:59.257346 3048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.257525 3034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:14:59.257648 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "8a8533fbd1d84474b30cfef24c59bdd1"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "f0d19db1611d7266eac4f4aec1a55c0f"
server_key_iv: "ded6a7b38c0e923817d77918cef1e268"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.257726 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:59.260957 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:59.261219 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:59.261356 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:59.261560 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:59.261893 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:59.261940 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.261966 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:59.261989 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.267668 3034 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:35167
I20260620 08:14:59.267745 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:35167 every 8 connection(s)
I20260620 08:14:59.268039 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:14:59.271099 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:59.268757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:46011 (local address 127.24.249.190:40007)
0620 08:14:59.268821 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:59.268823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:59.269015 (+ 192us) server_negotiation.cc:407] Connection header received
0620 08:14:59.269523 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:59.269524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:59.269531 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:59.269588 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:59.270839 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:59.270841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:59.270841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:59.270845 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:59.270862 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:59.270867 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:59.270923 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:59.271012 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 08:14:59.271028 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:14:59.271311 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:59.268721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40007 (local address 127.24.249.130:46011)
0620 08:14:59.268891 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:59.268931 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:59.269417 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:59.269658 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:59.269662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:59.269671 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:59.270677 (+ 1006us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:59.270678 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:59.270705 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:59.270731 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:59.270925 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:59.270930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:59.270933 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:14:59.271024 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 08:14:59.271084 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260620 08:14:59.271741 3162 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40007
I20260620 08:14:59.271823 3162 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:59.271986 3162 heartbeater.cc:511] Master 127.24.249.190:40007 requested a full tablet report, sending...
I20260620 08:14:59.272351 2843 ts_manager.cc:194] Registered new tserver with Master: 8a8533fbd1d84474b30cfef24c59bdd1 (127.24.249.130:35167)
I20260620 08:14:59.273523 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 3034
I20260620 08:14:59.273638 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:14:59.273931 25574 external_mini_cluster.cc:1428] Setting key dafbb79b4b37584cc0eede84eb8f7625
I20260620 08:14:59.274808 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40007
--builtin_ntp_servers=127.24.249.148:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 08:14:59.357775 3165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:14:59.358004 3165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:14:59.358026 3165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:14:59.359197 3165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:14:59.359236 3165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:14:59.359262 3165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:14:59.360704 3165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:40007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:14:59.360955 3165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:14:59.361289 3165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:14:59.366125 3171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:59.366155 3170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:14:59.366155 3173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:14:59.366400 3165 server_base.cc:1034] running on GCE node
I20260620 08:14:59.366647 3165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:14:59.366891 3165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:14:59.368074 3165 hybrid_clock.cc:648] HybridClock initialized: now 1781943299368053 us; error 32 us; skew 500 ppm
I20260620 08:14:59.369375 3165 webserver.cc:460] Webserver started at http://127.24.249.131:41997/ using document root <none> and password file <none>
I20260620 08:14:59.369624 3165 fs_manager.cc:359] Metadata directory not provided
I20260620 08:14:59.369693 3165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:14:59.369836 3165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:14:59.370924 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "03962fc3ea264664a8ae6cb6c024a137"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "d747bcd4fc7659f24eb33e26903ba194"
server_key_iv: "8ecef8b90d4db80c94fb5561b73e9e94"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.371294 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "03962fc3ea264664a8ae6cb6c024a137"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "d747bcd4fc7659f24eb33e26903ba194"
server_key_iv: "8ecef8b90d4db80c94fb5561b73e9e94"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.372612 3165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:14:59.373380 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.373554 3165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:14:59.373651 3165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "03962fc3ea264664a8ae6cb6c024a137"
format_stamp: "Formatted at 2026-06-20 08:14:59 on dist-test-slave-65w8"
server_key: "d747bcd4fc7659f24eb33e26903ba194"
server_key_iv: "8ecef8b90d4db80c94fb5561b73e9e94"
server_key_version: "encryptionkey@0"
I20260620 08:14:59.373780 3165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:14:59.377005 3165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:14:59.377257 3165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:14:59.377388 3165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:14:59.377590 3165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:14:59.377943 3165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:14:59.378010 3165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.378048 3165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:14:59.378074 3165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:14:59.383988 3165 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:41017
I20260620 08:14:59.384055 3292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:41017 every 8 connection(s)
I20260620 08:14:59.384382 3165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:14:59.387310 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:59.385090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:52015 (local address 127.24.249.190:40007)
0620 08:14:59.385135 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:59.385137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:59.385376 (+ 239us) server_negotiation.cc:407] Connection header received
0620 08:14:59.385806 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:59.385806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:59.385811 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:59.385915 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:59.387111 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:14:59.387113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:14:59.387113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:14:59.387118 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:14:59.387133 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:14:59.387137 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:14:59.387189 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 08:14:59.387238 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 08:14:59.387254 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:14:59.387478 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:14:59.385067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:40007 (local address 127.24.249.131:52015)
0620 08:14:59.385245 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:14:59.385284 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:14:59.385726 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:14:59.385980 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:14:59.385987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:14:59.385995 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:14:59.386958 (+ 963us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:14:59.386959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:14:59.386986 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:14:59.387013 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:14:59.387186 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:14:59.387188 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:14:59.387190 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:14:59.387241 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:14:59.387289 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260620 08:14:59.387964 3293 heartbeater.cc:347] Connected to a master server at 127.24.249.190:40007
I20260620 08:14:59.388038 3293 heartbeater.cc:464] Registering TS with master...
I20260620 08:14:59.388223 3293 heartbeater.cc:511] Master 127.24.249.190:40007 requested a full tablet report, sending...
I20260620 08:14:59.388582 2843 ts_manager.cc:194] Registered new tserver with Master: 03962fc3ea264664a8ae6cb6c024a137 (127.24.249.131:41017)
I20260620 08:14:59.393409 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 3165
I20260620 08:14:59.393484 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:14:59.393680 25574 external_mini_cluster.cc:1428] Setting key fd6d96fed65c73d86499140cba118bbe
I20260620 08:14:59.394541 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 08:14:59.396312 2887 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:14:59.395873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.24.249.190:40007)
0620 08:14:59.395908 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 08:14:59.395910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:14:59.396025 (+ 115us) server_negotiation.cc:407] Connection header received
0620 08:14:59.396071 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:14:59.396072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:14:59.396076 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:14:59.396124 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:14:59.396234 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52878: BlockingRecv error: recv got EOF from 127.0.0.1:52878 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 08:14:59.396427 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 2903
I20260620 08:14:59.401429 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 3034
I20260620 08:14:59.406045 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 3165
I20260620 08:14:59.410405 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 2812
2026-06-20T08:14:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (536 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:14:59 dist-test-slave-65w8 krb5kdc[3303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:14:59 dist-test-slave-65w8 krb5kdc[3303](info): set up 2 sockets
Jun 20 08:14:59 dist-test-slave-65w8 krb5kdc[3303](info): commencing operation
krb5kdc: starting...
W20260620 08:15:00.455067 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:15:00 dist-test-slave-65w8 krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:15:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:15:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:15:00.588776 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34493
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:15:00.677659 3316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:00.678011 3316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:00.678058 3316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:00.679476 3316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:15:00.679569 3316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:00.679596 3316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:00.679613 3316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:15:00.679636 3316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:15:00.681751 3316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:00.682091 3316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:00.682367 3316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:00.687542 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:00.687515 3321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:00.687604 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:00.687867 3316 server_base.cc:1034] running on GCE node
I20260620 08:15:00.688109 3316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:00.688376 3316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:00.689553 3316 hybrid_clock.cc:648] HybridClock initialized: now 1781943300689538 us; error 32 us; skew 500 ppm
Jun 20 08:15:00 dist-test-slave-65w8 krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943300, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:00.692480 3316 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:15:00.692992 3316 webserver.cc:460] Webserver started at http://127.24.249.190:40547/ using document root <none> and password file <none>
I20260620 08:15:00.693302 3316 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:00.693373 3316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:00.693475 3316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:15:00.694680 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "e89dc57dda5c4c0288b28a69801b325f"
format_stamp: "Formatted at 2026-06-20 08:15:00 on dist-test-slave-65w8"
server_key: "bc9ffc934788df45c882e5f403d22e63"
server_key_iv: "caac6c02f29953759c14c8216b503d18"
server_key_version: "encryptionkey@0"
I20260620 08:15:00.695003 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "e89dc57dda5c4c0288b28a69801b325f"
format_stamp: "Formatted at 2026-06-20 08:15:00 on dist-test-slave-65w8"
server_key: "bc9ffc934788df45c882e5f403d22e63"
server_key_iv: "caac6c02f29953759c14c8216b503d18"
server_key_version: "encryptionkey@0"
I20260620 08:15:00.696448 3316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:15:00.697235 3331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:00.697520 3316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:15:00.697629 3316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "e89dc57dda5c4c0288b28a69801b325f"
format_stamp: "Formatted at 2026-06-20 08:15:00 on dist-test-slave-65w8"
server_key: "bc9ffc934788df45c882e5f403d22e63"
server_key_iv: "caac6c02f29953759c14c8216b503d18"
server_key_version: "encryptionkey@0"
I20260620 08:15:00.697710 3316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:00.708263 3316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:00.710724 3316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:00.710896 3316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:00.715356 3316 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:34493
I20260620 08:15:00.715389 3383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:34493 every 8 connection(s)
I20260620 08:15:00.715744 3316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:15:00.716760 3384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:00.719197 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f: Bootstrap starting.
I20260620 08:15:00.719815 3384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:00.720113 3384 log.cc:824] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:00.720860 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f: No bootstrap required, opened a new log
I20260620 08:15:00.722620 3384 raft_consensus.cc:348] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89dc57dda5c4c0288b28a69801b325f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34493 } }
I20260620 08:15:00.722769 3384 raft_consensus.cc:374] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:00.722795 3384 raft_consensus.cc:729] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e89dc57dda5c4c0288b28a69801b325f, State: Initialized, Role: FOLLOWER
I20260620 08:15:00.722944 3384 consensus_queue.cc:260] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89dc57dda5c4c0288b28a69801b325f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34493 } }
I20260620 08:15:00.723022 3384 raft_consensus.cc:388] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:15:00.723063 3384 raft_consensus.cc:482] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:15:00.723114 3384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:15:00.723933 3384 raft_consensus.cc:504] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89dc57dda5c4c0288b28a69801b325f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34493 } }
I20260620 08:15:00.724020 3384 leader_election.cc:302] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e89dc57dda5c4c0288b28a69801b325f; no voters:
I20260620 08:15:00.724200 3384 leader_election.cc:288] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:15:00.724444 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 3316
I20260620 08:15:00.724560 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:15:00.724758 25574 external_mini_cluster.cc:1428] Setting key 96b5d6b96da2f56fe2a8cfde29f80449
I20260620 08:15:00.724321 3387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:15:00.725152 3387 raft_consensus.cc:686] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [term 1 LEADER]: Becoming Leader. State: Replica: e89dc57dda5c4c0288b28a69801b325f, State: Running, Role: LEADER
I20260620 08:15:00.725184 3384 sys_catalog.cc:564] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:15:00.725307 3387 consensus_queue.cc:237] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89dc57dda5c4c0288b28a69801b325f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34493 } }
I20260620 08:15:00.725656 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e89dc57dda5c4c0288b28a69801b325f. Latest consensus state: current_term: 1 leader_uuid: "e89dc57dda5c4c0288b28a69801b325f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89dc57dda5c4c0288b28a69801b325f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34493 } } }
I20260620 08:15:00.725668 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e89dc57dda5c4c0288b28a69801b325f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e89dc57dda5c4c0288b28a69801b325f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34493 } } }
I20260620 08:15:00.725850 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [sys.catalog]: This master's current role is: LEADER
I20260620 08:15:00.725899 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f [sys.catalog]: This master's current role is: LEADER
I20260620 08:15:00.726604 3398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:15:00.727197 3398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:15:00.729250 3398 catalog_manager.cc:1347] Generated new cluster ID: 41c999b2ccfa4404a706cffc943c559d
I20260620 08:15:00.729305 3398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:15:00 dist-test-slave-65w8 krb5kdc[3303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:00.735184 3398 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:15:00.735877 3398 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:15:00.736835 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:00.725492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.24.249.190:34493)
0620 08:15:00.726080 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:00.726089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:00.726122 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:15:00.726240 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:00.726247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:00.726304 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:00.727255 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:00.727362 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:00.728755 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:00.729838 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:00.730291 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:00.730427 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:00.732872 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:00.732876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:00.732877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:00.732889 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:00.735077 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:00.735543 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:00.735546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:00.735547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:00.735642 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:00.735873 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:00.735874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:00.735875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:00.736185 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:00.736366 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:00.736468 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 08:15:00.736629 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":87,"threads_started":1}
I20260620 08:15:00.742645 3398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f: Generated new TSK 0
I20260620 08:15:00.742833 3398 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:15:00.777423 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34493
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:15:00.875953 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:00.876209 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:00.876251 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:00.877424 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:00.877489 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:00.877539 3410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:15:00.879012 3410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:00.879277 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:00.879503 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:00.885106 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:00.885097 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:00.885136 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:00.885308 3410 server_base.cc:1034] running on GCE node
I20260620 08:15:00.885520 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:00.885747 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:00.887077 3410 hybrid_clock.cc:648] HybridClock initialized: now 1781943300886892 us; error 44 us; skew 500 ppm
Jun 20 08:15:00 dist-test-slave-65w8 krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943300, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:00.889842 3410 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:15:00.890363 3410 webserver.cc:460] Webserver started at http://127.24.249.129:41785/ using document root <none> and password file <none>
I20260620 08:15:00.890594 3410 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:00.890661 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:00.890816 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:15:00.891913 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "688866f210154403af6504786d571ce2"
format_stamp: "Formatted at 2026-06-20 08:15:00 on dist-test-slave-65w8"
server_key: "b0a47d01966670022c7df55e857467ad"
server_key_iv: "1fe13564108a0b7ee833f753febdc4b4"
server_key_version: "encryptionkey@0"
I20260620 08:15:00.892285 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "688866f210154403af6504786d571ce2"
format_stamp: "Formatted at 2026-06-20 08:15:00 on dist-test-slave-65w8"
server_key: "b0a47d01966670022c7df55e857467ad"
server_key_iv: "1fe13564108a0b7ee833f753febdc4b4"
server_key_version: "encryptionkey@0"
I20260620 08:15:00.893775 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:15:00.894665 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:00.894891 3410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:15:00.895005 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "688866f210154403af6504786d571ce2"
format_stamp: "Formatted at 2026-06-20 08:15:00 on dist-test-slave-65w8"
server_key: "b0a47d01966670022c7df55e857467ad"
server_key_iv: "1fe13564108a0b7ee833f753febdc4b4"
server_key_version: "encryptionkey@0"
I20260620 08:15:00.895094 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:00.904402 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:00.906805 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:00.906997 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:00.907290 3410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:15:00.907620 3410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:15:00.907675 3410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:00.907720 3410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:15:00.907745 3410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:00.913386 3410 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:39707
I20260620 08:15:00.913463 3538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:39707 every 8 connection(s)
I20260620 08:15:00.913798 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:15:00 dist-test-slave-65w8 krb5kdc[3303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943300, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:00.923522 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 3410
I20260620 08:15:00.923571 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:00.914631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:34449 (local address 127.24.249.190:34493)
0620 08:15:00.914754 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:00.914756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:00.914953 (+ 197us) server_negotiation.cc:407] Connection header received
0620 08:15:00.915491 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:00.915492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:00.915505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:00.915576 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:00.916600 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:00.917109 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:00.918412 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:00.918626 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:00.918704 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:00.920872 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:00.920874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:00.920874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:00.920882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:00.922152 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:00.922564 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:00.922565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:00.922566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:00.922625 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:00.922880 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:00.922881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:00.922881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:00.922998 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:00.923074 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:00.923370 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 08:15:00.923462 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 08:15:00.923696 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:15:00.923776 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:00.914562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34493 (local address 127.24.249.129:34449)
0620 08:15:00.914836 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:00.914870 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:00.915378 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:00.915656 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:00.915660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:00.916297 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:15:00.916496 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:00.916501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:00.917261 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:00.917263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:00.918321 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:00.918323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:00.918712 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:00.918713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:00.918844 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:00.919328 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:15:00.919336 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:15:00.920772 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:15:00.922247 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:00.922254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:00.922256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:00.922474 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:00.922692 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:00.922693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:00.922694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:00.922802 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:00.923074 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:15:00.923082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:15:00.923224 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 08:15:00.923336 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 08:15:00.923492 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260620 08:15:00.923923 25574 external_mini_cluster.cc:1428] Setting key 9a8e572bbc4c5a280657df74af5e4d87
I20260620 08:15:00.924487 3539 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34493
I20260620 08:15:00.924624 3539 heartbeater.cc:464] Registering TS with master...
I20260620 08:15:00.924872 3539 heartbeater.cc:511] Master 127.24.249.190:34493 requested a full tablet report, sending...
I20260620 08:15:00.925453 3347 ts_manager.cc:194] Registered new tserver with Master: 688866f210154403af6504786d571ce2 (127.24.249.129:39707)
I20260620 08:15:00.926580 3347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:34449
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:15:00.951169 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34493
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:15:01.035774 3544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:01.036033 3544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:01.036060 3544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:01.037319 3544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:01.037361 3544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:01.037387 3544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:15:01.038863 3544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:01.039124 3544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:01.039366 3544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:01.044512 3549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:01.044512 3552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:01.044569 3550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:01.044690 3544 server_base.cc:1034] running on GCE node
I20260620 08:15:01.045070 3544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:01.045341 3544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:01.046525 3544 hybrid_clock.cc:648] HybridClock initialized: now 1781943301046512 us; error 33 us; skew 500 ppm
Jun 20 08:15:01 dist-test-slave-65w8 krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943301, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:01.049203 3544 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:15:01.049666 3544 webserver.cc:460] Webserver started at http://127.24.249.130:34915/ using document root <none> and password file <none>
I20260620 08:15:01.049983 3544 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:01.050051 3544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:01.050208 3544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:15:01.051270 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "c39eec3c1c2945f5bcd12622f29bdbd4"
format_stamp: "Formatted at 2026-06-20 08:15:01 on dist-test-slave-65w8"
server_key: "d95630b08e6ceaf34576a5b7d96d4d0b"
server_key_iv: "34b9d91ca39f6af75360582041642d0b"
server_key_version: "encryptionkey@0"
I20260620 08:15:01.051640 3544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "c39eec3c1c2945f5bcd12622f29bdbd4"
format_stamp: "Formatted at 2026-06-20 08:15:01 on dist-test-slave-65w8"
server_key: "d95630b08e6ceaf34576a5b7d96d4d0b"
server_key_iv: "34b9d91ca39f6af75360582041642d0b"
server_key_version: "encryptionkey@0"
I20260620 08:15:01.053068 3544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:15:01.053959 3559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:01.054205 3544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:15:01.054314 3544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "c39eec3c1c2945f5bcd12622f29bdbd4"
format_stamp: "Formatted at 2026-06-20 08:15:01 on dist-test-slave-65w8"
server_key: "d95630b08e6ceaf34576a5b7d96d4d0b"
server_key_iv: "34b9d91ca39f6af75360582041642d0b"
server_key_version: "encryptionkey@0"
I20260620 08:15:01.054405 3544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:01.077517 3544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:01.079681 3544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:01.079841 3544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:01.080097 3544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:15:01.080438 3544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:15:01.080495 3544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:01.080540 3544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:15:01.080567 3544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:01.086568 3544 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:38231
I20260620 08:15:01.086664 3672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:38231 every 8 connection(s)
I20260620 08:15:01.086977 3544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:15:01.096882 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 3544
I20260620 08:15:01.096974 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:15:01.097211 25574 external_mini_cluster.cc:1428] Setting key f37c1a9aa446c0d96f5c8f9df3476721
Jun 20 08:15:01 dist-test-slave-65w8 krb5kdc[3303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943301, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:01.105305 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.087769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:56267 (local address 127.24.249.190:34493)
0620 08:15:01.087841 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.087843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.088189 (+ 346us) server_negotiation.cc:407] Connection header received
0620 08:15:01.088711 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.088712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.088724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.088778 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:01.090006 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.090539 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.091551 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.091751 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.091835 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.101772 (+ 9937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:01.101774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:01.101774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:01.101782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:01.103582 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:01.104005 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:01.104006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:01.104007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:01.104072 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:01.104374 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:01.104375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:01.104375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:01.104538 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:01.104619 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.105071 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.105178 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:15:01.105340 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.087798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34493 (local address 127.24.249.130:56267)
0620 08:15:01.088032 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.088076 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:01.088600 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:01.088852 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:01.088855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:01.089448 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:15:01.089816 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.089822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.090643 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.090644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.091448 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.091450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.091846 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.091847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.091967 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.099621 (+ 7654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:15:01.099638 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:15:01.101654 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:15:01.103681 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:01.103692 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:01.103694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:01.103924 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:01.104169 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:01.104170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:01.104170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:01.104295 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:01.104779 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:15:01.104785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:15:01.104921 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 08:15:01.105022 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 08:15:01.105164 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260620 08:15:01.106115 3673 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34493
I20260620 08:15:01.106240 3673 heartbeater.cc:464] Registering TS with master...
I20260620 08:15:01.106455 3673 heartbeater.cc:511] Master 127.24.249.190:34493 requested a full tablet report, sending...
I20260620 08:15:01.106976 3347 ts_manager.cc:194] Registered new tserver with Master: c39eec3c1c2945f5bcd12622f29bdbd4 (127.24.249.130:38231)
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:15:01.107465 3347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:56267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:15:01.125337 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34493
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:15:01.209249 3678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:01.209483 3678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:01.209510 3678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:01.211054 3678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:01.211107 3678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:01.211135 3678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:15:01.212842 3678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:01.213120 3678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:01.213392 3678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:01.218657 3686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:01.218658 3684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:01.218751 3683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:01.219105 3678 server_base.cc:1034] running on GCE node
I20260620 08:15:01.219291 3678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:01.219563 3678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:01.220736 3678 hybrid_clock.cc:648] HybridClock initialized: now 1781943301220714 us; error 35 us; skew 500 ppm
Jun 20 08:15:01 dist-test-slave-65w8 krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943301, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:01.223492 3678 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:15:01.224027 3678 webserver.cc:460] Webserver started at http://127.24.249.131:36473/ using document root <none> and password file <none>
I20260620 08:15:01.224287 3678 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:01.224354 3678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:01.224500 3678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:15:01.225791 3678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "22603820bfc449478d0735543b643b91"
format_stamp: "Formatted at 2026-06-20 08:15:01 on dist-test-slave-65w8"
server_key: "8adf784a9c7b550b3864676d83e7feb6"
server_key_iv: "a0232187dbd73839d2add5147ecebc69"
server_key_version: "encryptionkey@0"
I20260620 08:15:01.226320 3678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "22603820bfc449478d0735543b643b91"
format_stamp: "Formatted at 2026-06-20 08:15:01 on dist-test-slave-65w8"
server_key: "8adf784a9c7b550b3864676d83e7feb6"
server_key_iv: "a0232187dbd73839d2add5147ecebc69"
server_key_version: "encryptionkey@0"
I20260620 08:15:01.227797 3678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:15:01.228578 3693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:01.228804 3678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:15:01.228904 3678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "22603820bfc449478d0735543b643b91"
format_stamp: "Formatted at 2026-06-20 08:15:01 on dist-test-slave-65w8"
server_key: "8adf784a9c7b550b3864676d83e7feb6"
server_key_iv: "a0232187dbd73839d2add5147ecebc69"
server_key_version: "encryptionkey@0"
I20260620 08:15:01.228991 3678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:01.261457 3678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:01.263434 3678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:01.263614 3678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:01.263854 3678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:15:01.264173 3678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:15:01.264235 3678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:01.264277 3678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:15:01.264305 3678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:01.269942 3678 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:39565
I20260620 08:15:01.270009 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:39565 every 8 connection(s)
I20260620 08:15:01.270354 3678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:15:01.271107 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 3678
I20260620 08:15:01.271217 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:15:01.271416 25574 external_mini_cluster.cc:1428] Setting key a0f55260b6517f21124e4d47a9cdd49c
Jun 20 08:15:01 dist-test-slave-65w8 krb5kdc[3303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943301, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:01.279536 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.271211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:34359 (local address 127.24.249.190:34493)
0620 08:15:01.271268 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.271271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.271459 (+ 188us) server_negotiation.cc:407] Connection header received
0620 08:15:01.272038 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.272039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.272051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.272106 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:01.273037 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.273537 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.274556 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.274773 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.274861 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.276860 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:01.276862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:01.276862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:01.276870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:01.278190 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:01.278595 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:01.278596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:01.278597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:01.278654 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:01.278915 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:01.278916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:01.278916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:01.279026 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:01.279102 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.279366 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.279442 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:15:01.279731 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.271173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34493 (local address 127.24.249.131:34359)
0620 08:15:01.271322 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.271362 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:01.271937 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:01.272179 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:01.272193 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:01.272742 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:15:01.272928 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.272941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.273642 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.273643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.274461 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.274463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.274869 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.274870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.274991 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.275450 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:15:01.275458 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:15:01.276764 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:15:01.278287 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:01.278293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:01.278296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:01.278505 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:01.278737 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:01.278738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:01.278738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:01.278838 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:01.279093 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:15:01.279098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:15:01.279238 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 08:15:01.279346 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 08:15:01.279499 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260620 08:15:01.280246 3807 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34493
I20260620 08:15:01.280321 3807 heartbeater.cc:464] Registering TS with master...
I20260620 08:15:01.280455 3807 heartbeater.cc:511] Master 127.24.249.190:34493 requested a full tablet report, sending...
I20260620 08:15:01.280799 3347 ts_manager.cc:194] Registered new tserver with Master: 22603820bfc449478d0735543b643b91 (127.24.249.131:39565)
I20260620 08:15:01.281117 3347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:34359
I20260620 08:15:01.283017 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:15:01.290184 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.284353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.24.249.190:34493)
0620 08:15:01.284408 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.284409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.284518 (+ 109us) server_negotiation.cc:407] Connection header received
0620 08:15:01.284553 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.284554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.284562 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.284610 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:01.285121 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.285582 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.286336 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.286512 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.286567 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.287590 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:01.287591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:01.287591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:01.287597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:01.288916 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:01.289293 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:01.289294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:01.289294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:01.289343 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:01.289573 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:01.289574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:01.289574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:01.289742 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:01.289821 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.290057 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.290115 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:15:01.291860 3347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:15:01.293558 3347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:15:01.300828 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.296220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39707 (local address 127.0.0.1:60654)
0620 08:15:01.296504 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.296515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:01.296565 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:01.296914 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:01.296915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:01.296917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:01.296974 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.296976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.298399 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.298400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.299528 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.299530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.300521 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.300522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.300602 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.300616 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:15:01.300692 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:15:01.300743 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
I20260620 08:15:01.300889 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.296555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.24.249.129:39707)
0620 08:15:01.296707 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.296711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.296720 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:15:01.296792 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.296795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.296810 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.297046 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:01.297116 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.298275 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.299653 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.300430 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.300514 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.300602 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.300710 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.300803 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260620 08:15:01.301314 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.296157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:38231 (local address 127.0.0.1:43500)
0620 08:15:01.296287 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.296332 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:01.296398 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:01.296814 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:01.296818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:01.296821 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:01.296896 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.296898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.298100 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.298101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.299528 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.299530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.301089 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.301090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.301152 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.301156 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:15:01.301207 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 08:15:01.301248 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":33,"thread_start_us":80,"threads_started":1}
I20260620 08:15:01.301606 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.296292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.24.249.130:38231)
0620 08:15:01.296592 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.296597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.296608 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:15:01.296649 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.296651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.296668 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.296750 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:01.296982 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.297994 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.299653 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.300430 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.300514 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.300573 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.301460 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.301517 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":90,"threads_started":1}
I20260620 08:15:01.301654 3473 tablet_service.cc:1467] Processing CreateTablet for tablet 8255dfce18354bb8b83d4ac655e0bd76 (DEFAULT_TABLE table=test-table [id=1b5eeafba6764d02be689ba2dd39d705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:15:01.302012 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.296466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.24.249.131:39565)
0620 08:15:01.296735 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.296739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.296972 (+ 233us) server_negotiation.cc:407] Connection header received
0620 08:15:01.297012 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.297014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.297029 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.297102 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:01.297198 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.298306 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.300438 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.301554 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.301631 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.301687 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.301890 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.301932 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":150,"threads_started":1}
I20260620 08:15:01.302026 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.296377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39565 (local address 127.0.0.1:37504)
0620 08:15:01.296860 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.296870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:01.296915 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:01.297100 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:01.297101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:01.297103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:01.297137 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.297138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.299636 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.299637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.300367 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.300369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.301642 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.301643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.301774 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.301779 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:15:01.301904 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 08:15:01.301954 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":70,"threads_started":1}
I20260620 08:15:01.302167 3473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8255dfce18354bb8b83d4ac655e0bd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:01.302482 3607 tablet_service.cc:1467] Processing CreateTablet for tablet 8255dfce18354bb8b83d4ac655e0bd76 (DEFAULT_TABLE table=test-table [id=1b5eeafba6764d02be689ba2dd39d705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:15:01.302791 3607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8255dfce18354bb8b83d4ac655e0bd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:01.305688 3824 tablet_bootstrap.cc:492] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2: Bootstrap starting.
I20260620 08:15:01.306005 3825 tablet_bootstrap.cc:492] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Bootstrap starting.
I20260620 08:15:01.306279 3741 tablet_service.cc:1467] Processing CreateTablet for tablet 8255dfce18354bb8b83d4ac655e0bd76 (DEFAULT_TABLE table=test-table [id=1b5eeafba6764d02be689ba2dd39d705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:15:01.306480 3824 tablet_bootstrap.cc:654] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:01.306586 3741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8255dfce18354bb8b83d4ac655e0bd76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:01.306690 3825 tablet_bootstrap.cc:654] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:01.307099 3824 log.cc:824] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:01.307315 3825 log.cc:824] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:01.307904 3824 tablet_bootstrap.cc:492] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2: No bootstrap required, opened a new log
I20260620 08:15:01.307960 3824 ts_tablet_manager.cc:1397] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:15:01.309229 3828 tablet_bootstrap.cc:492] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Bootstrap starting.
I20260620 08:15:01.309293 3824 raft_consensus.cc:348] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.309507 3824 raft_consensus.cc:374] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:01.309557 3824 raft_consensus.cc:729] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688866f210154403af6504786d571ce2, State: Initialized, Role: FOLLOWER
I20260620 08:15:01.309687 3824 consensus_queue.cc:260] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.309917 3828 tablet_bootstrap.cc:654] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:01.309965 3539 heartbeater.cc:503] Master 127.24.249.190:34493 was elected leader, sending a full tablet report...
I20260620 08:15:01.309978 3824 ts_tablet_manager.cc:1428] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:15:01.310559 3828 log.cc:824] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:01.310770 3825 tablet_bootstrap.cc:492] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: No bootstrap required, opened a new log
I20260620 08:15:01.310848 3825 ts_tablet_manager.cc:1397] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:15:01.311298 3828 tablet_bootstrap.cc:492] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: No bootstrap required, opened a new log
I20260620 08:15:01.311352 3828 ts_tablet_manager.cc:1397] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:15:01.312438 3825 raft_consensus.cc:348] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.312585 3825 raft_consensus.cc:374] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:01.312662 3825 raft_consensus.cc:729] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39eec3c1c2945f5bcd12622f29bdbd4, State: Initialized, Role: FOLLOWER
I20260620 08:15:01.312803 3825 consensus_queue.cc:260] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.312928 3828 raft_consensus.cc:348] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.313081 3825 ts_tablet_manager.cc:1428] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:15:01.313088 3828 raft_consensus.cc:374] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:01.313130 3828 raft_consensus.cc:729] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22603820bfc449478d0735543b643b91, State: Initialized, Role: FOLLOWER
I20260620 08:15:01.313115 3673 heartbeater.cc:503] Master 127.24.249.190:34493 was elected leader, sending a full tablet report...
I20260620 08:15:01.313246 3828 consensus_queue.cc:260] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.313517 3828 ts_tablet_manager.cc:1428] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:15:01.313637 3807 heartbeater.cc:503] Master 127.24.249.190:34493 was elected leader, sending a full tablet report...
W20260620 08:15:01.337568 3674 tablet.cc:2369] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:15:01.377512 3829 raft_consensus.cc:482] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:15:01.377676 3829 raft_consensus.cc:504] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.378103 3829 leader_election.cc:288] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39eec3c1c2945f5bcd12622f29bdbd4 (127.24.249.130:38231), 22603820bfc449478d0735543b643b91 (127.24.249.131:39565)
I20260620 08:15:01.381131 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.378460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39565 (local address 127.24.249.129:46143)
0620 08:15:01.378641 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.378652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:01.378704 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:01.378840 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:01.378841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:01.378842 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:01.378929 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.378930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.379582 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.379583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.380317 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.380319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.380871 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.380872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.380942 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.380945 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:15:01.380988 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 08:15:01.381038 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260620 08:15:01.381139 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.378553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:46143 (local address 127.24.249.131:39565)
0620 08:15:01.378614 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.378616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.378725 (+ 109us) server_negotiation.cc:407] Connection header received
0620 08:15:01.378753 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.378754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.378765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.378823 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:01.378999 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.379499 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.380412 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.380783 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.380840 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.380872 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.381015 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.381063 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:15:01.381175 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.378417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:45171 (local address 127.24.249.130:38231)
0620 08:15:01.378484 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.378486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.378496 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:15:01.378526 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.378527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.378539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.378616 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:01.378841 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.379397 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.380410 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.380790 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.380879 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.380887 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.381060 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.381101 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:15:01.381209 3541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.378319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:38231 (local address 127.24.249.129:45171)
0620 08:15:01.378385 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.378396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:01.378465 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:01.378635 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:01.378636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:01.378638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:01.378748 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.378751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.379482 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.379483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.380315 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:01.380317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.380896 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.380897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:01.380965 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.380968 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:15:01.381043 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:15:01.381078 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 08:15:01.381788 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8255dfce18354bb8b83d4ac655e0bd76" candidate_uuid: "688866f210154403af6504786d571ce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" is_pre_election: true
I20260620 08:15:01.381788 3761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8255dfce18354bb8b83d4ac655e0bd76" candidate_uuid: "688866f210154403af6504786d571ce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22603820bfc449478d0735543b643b91" is_pre_election: true
I20260620 08:15:01.381968 3627 raft_consensus.cc:2454] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688866f210154403af6504786d571ce2 in term 0.
I20260620 08:15:01.381968 3761 raft_consensus.cc:2454] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 688866f210154403af6504786d571ce2 in term 0.
I20260620 08:15:01.382189 3429 leader_election.cc:302] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22603820bfc449478d0735543b643b91, 688866f210154403af6504786d571ce2; no voters:
I20260620 08:15:01.382321 3829 raft_consensus.cc:2781] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:15:01.382377 3829 raft_consensus.cc:482] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:15:01.382408 3829 raft_consensus.cc:3037] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:15:01.383114 3829 raft_consensus.cc:504] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.383293 3829 leader_election.cc:288] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [CANDIDATE]: Term 1 election: Requested vote from peers c39eec3c1c2945f5bcd12622f29bdbd4 (127.24.249.130:38231), 22603820bfc449478d0735543b643b91 (127.24.249.131:39565)
I20260620 08:15:01.383414 3627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8255dfce18354bb8b83d4ac655e0bd76" candidate_uuid: "688866f210154403af6504786d571ce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4"
I20260620 08:15:01.383464 3761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8255dfce18354bb8b83d4ac655e0bd76" candidate_uuid: "688866f210154403af6504786d571ce2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22603820bfc449478d0735543b643b91"
I20260620 08:15:01.383496 3627 raft_consensus.cc:3037] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:15:01.383530 3761 raft_consensus.cc:3037] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:15:01.384495 3627 raft_consensus.cc:2454] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688866f210154403af6504786d571ce2 in term 1.
I20260620 08:15:01.384496 3761 raft_consensus.cc:2454] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 688866f210154403af6504786d571ce2 in term 1.
I20260620 08:15:01.384682 3429 leader_election.cc:302] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22603820bfc449478d0735543b643b91, 688866f210154403af6504786d571ce2; no voters:
I20260620 08:15:01.384789 3829 raft_consensus.cc:2781] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:15:01.384943 3829 raft_consensus.cc:686] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 1 LEADER]: Becoming Leader. State: Replica: 688866f210154403af6504786d571ce2, State: Running, Role: LEADER
I20260620 08:15:01.385046 3829 consensus_queue.cc:237] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } }
I20260620 08:15:01.385972 3347 catalog_manager.cc:5495] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 688866f210154403af6504786d571ce2 (127.24.249.129). New cstate: current_term: 1 leader_uuid: "688866f210154403af6504786d571ce2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "688866f210154403af6504786d571ce2" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 39707 } health_report { overall_health: HEALTHY } } }
I20260620 08:15:01.392457 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.390080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.24.249.129:39707)
0620 08:15:01.390161 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:01.390162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:01.390171 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:15:01.390197 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:01.390198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:01.390214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:01.390277 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:15:01.390453 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.390943 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.391707 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:01.391885 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:01.391944 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:01.392116 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:15:01.392209 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:15:01.392277 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:01.392318 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 08:15:01.392337 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 08:15:01.394194 3627 raft_consensus.cc:1261] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 1 FOLLOWER]: Refusing update from remote peer 688866f210154403af6504786d571ce2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:15:01.394212 3761 raft_consensus.cc:1261] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 1 FOLLOWER]: Refusing update from remote peer 688866f210154403af6504786d571ce2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:15:01.394536 3829 consensus_queue.cc:1035] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22603820bfc449478d0735543b643b91" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:15:01.394760 3834 consensus_queue.cc:1035] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39eec3c1c2945f5bcd12622f29bdbd4" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:15:01.396943 3837 mvcc.cc:205] Tried to move back new op lower bound from 7298839762507665408 to 7298839762473783296. Current Snapshot: MvccSnapshot[applied={T|T < 7298839762507665408}]
I20260620 08:15:01.397146 3839 mvcc.cc:205] Tried to move back new op lower bound from 7298839762507665408 to 7298839762473783296. Current Snapshot: MvccSnapshot[applied={T|T < 7298839762507665408}]
I20260620 08:15:01.398149 3838 mvcc.cc:205] Tried to move back new op lower bound from 7298839762507665408 to 7298839762473783296. Current Snapshot: MvccSnapshot[applied={T|T < 7298839762507665408 or (T in {7298839762507665408})}]
I20260620 08:15:01.399237 3348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:15:01.399325 3348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:15:01.400092 3348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e89dc57dda5c4c0288b28a69801b325f: Sending DeleteTablet for 3 replicas of tablet 8255dfce18354bb8b83d4ac655e0bd76
I20260620 08:15:01.400458 3607 tablet_service.cc:1514] Processing DeleteTablet for tablet 8255dfce18354bb8b83d4ac655e0bd76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:15:01 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:43500
I20260620 08:15:01.400455 3473 tablet_service.cc:1514] Processing DeleteTablet for tablet 8255dfce18354bb8b83d4ac655e0bd76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:15:01 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:60654
I20260620 08:15:01.400700 3848 tablet_replica.cc:330] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2: stopping tablet replica
I20260620 08:15:01.400705 3741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8255dfce18354bb8b83d4ac655e0bd76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:15:01 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:37504
I20260620 08:15:01.400799 3848 raft_consensus.cc:2229] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:15:01.400892 3848 raft_consensus.cc:2258] T 8255dfce18354bb8b83d4ac655e0bd76 P 688866f210154403af6504786d571ce2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:15:01.401088 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 3410
I20260620 08:15:01.401569 3847 tablet_replica.cc:330] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: stopping tablet replica
I20260620 08:15:01.401654 3847 raft_consensus.cc:2229] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:15:01.401721 3847 raft_consensus.cc:2258] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:15:01.402921 3849 tablet_replica.cc:330] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: stopping tablet replica
I20260620 08:15:01.403023 3849 raft_consensus.cc:2229] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:15:01.403133 3849 raft_consensus.cc:2258] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:15:01.403497 3849 ts_tablet_manager.cc:1905] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:15:01.403560 3847 ts_tablet_manager.cc:1905] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:15:01.404949 3849 ts_tablet_manager.cc:1918] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:15:01.405037 3849 log.cc:1196] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/8255dfce18354bb8b83d4ac655e0bd76
I20260620 08:15:01.405321 3849 ts_tablet_manager.cc:1939] T 8255dfce18354bb8b83d4ac655e0bd76 P 22603820bfc449478d0735543b643b91: Deleting consensus metadata
I20260620 08:15:01.405534 3847 ts_tablet_manager.cc:1918] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:15:01.405611 3847 log.cc:1196] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/8255dfce18354bb8b83d4ac655e0bd76
I20260620 08:15:01.405740 3335 catalog_manager.cc:4841] TS 22603820bfc449478d0735543b643b91 (127.24.249.131:39565): tablet 8255dfce18354bb8b83d4ac655e0bd76 (table test-table [id=1b5eeafba6764d02be689ba2dd39d705]) successfully deleted
I20260620 08:15:01.405946 3847 ts_tablet_manager.cc:1939] T 8255dfce18354bb8b83d4ac655e0bd76 P c39eec3c1c2945f5bcd12622f29bdbd4: Deleting consensus metadata
I20260620 08:15:01.406328 3335 catalog_manager.cc:4841] TS c39eec3c1c2945f5bcd12622f29bdbd4 (127.24.249.130:38231): tablet 8255dfce18354bb8b83d4ac655e0bd76 (table test-table [id=1b5eeafba6764d02be689ba2dd39d705]) successfully deleted
W20260620 08:15:01.408066 3332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:39707 (error 108)
I20260620 08:15:01.408567 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 3544
I20260620 08:15:01.408587 3821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:01.408444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:39707 (local address 127.0.0.1:60670)
0620 08:15:01.408513 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:01.408553 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:39707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:15:01.408702 3332 catalog_manager.cc:4584] TS 688866f210154403af6504786d571ce2 (127.24.249.129:39707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8255dfce18354bb8b83d4ac655e0bd76: Network error: Client connection negotiation failed: client connection to 127.24.249.129:39707: connect: Connection refused (error 111)
I20260620 08:15:01.413707 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 3678
I20260620 08:15:01.418671 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 3316
2026-06-20T08:15:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2011 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:15:01 dist-test-slave-65w8 krb5kdc[3853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:15:01 dist-test-slave-65w8 krb5kdc[3853](info): set up 2 sockets
Jun 20 08:15:01 dist-test-slave-65w8 krb5kdc[3853](info): commencing operation
krb5kdc: starting...
W20260620 08:15:02.491160 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:15:02 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:15:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:15:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/rangeradmin_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/rangeradmin_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/rangerlookup_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/rangerlookup_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
I20260620 08:15:02.672556 25574 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5EkY3n/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/postgres -l logfile start
2026-06-20 08:15:05.533 UTC [3883] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 08:15:05.533 UTC [3883] LOG: listening on IPv4 address "127.24.249.148", port 36595
2026-06-20 08:15:05.538 UTC [3883] LOG: listening on Unix socket "/tmp/.s.PGSQL.36595"
2026-06-20 08:15:05.562 UTC [3886] LOG: database system was shut down at 2026-06-20 08:15:03 UTC
2026-06-20 08:15:05.566 UTC [3883] LOG: database system is ready to accept connections
I20260620 08:15:06.551079 25574 mini_postgres.cc:96] Postgres bound to 36595
2026-06-20 08:15:06.557 UTC [3894] FATAL: database "slave" does not exist
127.24.249.148:36595 - accepting connections
I20260620 08:15:06.558247 25574 mini_ranger.cc:161] Starting Ranger...
I20260620 08:15:06.573225 25574 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 08:15:07.136940 25574 mini_ranger.cc:87] Created ranger Postgres database
I20260620 08:15:07.137041 25574 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-admin
2026-06-20 08:15:07,485 [I] DB FLAVOR :POSTGRES
2026-06-20 08:15:07,486 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:15:07,486 [I] Checking connection..
2026-06-20 08:15:07,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 08:15:07,777 [I] Checking connection passed.
2026-06-20 08:15:07,778 [I] --------- Verifying version history table ---------
2026-06-20 08:15:07,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:15:08,024 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:15:08,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:15:08,267 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:15:08,268 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 08:15:08,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 08:15:08.528 UTC [3958] WARNING: there is no transaction in progress
2026-06-20 08:15:08,534 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 08:15:08,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:15:08,791 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 08:15:08,792 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:15:08,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 08:15:09,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 08:15:09,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-65w8.c.gcp-upstream.internal','N') ;"
2026-06-20 08:15:09,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:15:09,814 [I] Table x_portal_user does not exist in database ranger
2026-06-20 08:15:09,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:15:10,080 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 08:15:10,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:15:10,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:15:10,588 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 08:15:10,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 08:15:11.604 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.622 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.639 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.656 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.675 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.698 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.762 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.770 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.786 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.798 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.811 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.828 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.839 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.846 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.853 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:11.861 UTC [4066] WARNING: there is no transaction in progress
2026-06-20 08:15:12,666 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 08:15:12,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 08:15:12,928 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 08:15:12,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:15:13,173 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 08:15:13,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:15:13,445 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 08:15:13,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:15:13,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:15:13,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-65w8.c.gcp-upstream.internal';"
2026-06-20 08:15:14,172 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 08:15:14,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 08:15:14,433 [I] DB_PATCHES have already been applied
I20260620 08:15:14.439553 25574 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-admin:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5EkY3n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 08:15:14.439618 25574 mini_ranger.cc:193] Using host: 127.24.249.148
I20260620 08:15:14.443116 25574 mini_ranger.cc:227] Ranger admin URL: http://127.24.249.148:54733
Jun 20, 2026 8:15:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:15:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:15:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:15:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:15:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:15:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 08:15:15.560951 25574 mini_ranger.cc:160] Time spent starting Ranger: real 9.003s user 0.000s sys 0.010s
Jun 20, 2026 8:15:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:15:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 8:15:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:15:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:15:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 08:15:39.259262 25574 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/rangerkms_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/rangerkms_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.249.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.249.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 08:15:39 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 08:15:39.342635 25574 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 08:15:39.352180 25574 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 08:15:40.245291 25574 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 08:15:40.245378 25574 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms
I20260620 08:15:40.245401 25574 mini_ranger_kms.cc:227] Using postgres at 127.24.249.148:36595
2026-06-20 08:15:40,637 [I] DB FLAVOR :POSTGRES
2026-06-20 08:15:40,637 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:15:40,637 [I] Checking connection
2026-06-20 08:15:40,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 08:15:40,938 [I] connection success
2026-06-20 08:15:40,938 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 08:15:40,938 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 08:15:40,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 08:15:41,194 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 08:15:41,194 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:15:41,195 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 08:15:41,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:36595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 08:15:41,491 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 08:15:42.015380 25574 mini_ranger_kms.cc:310] Created kms service
I20260620 08:15:42.241503 25574 mini_ranger_kms.cc:326] Created kudu user
I20260620 08:15:42.283754 25574 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 08:15:42 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:15:42.724203 25574 mini_ranger_kms.cc:384] Added ranger policy
I20260620 08:15:42.724347 25574 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms:/tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 08:15:42.724373 25574 mini_ranger_kms.cc:240] Using host: 127.24.249.148
I20260620 08:15:42.728255 25574 mini_ranger_kms.cc:276] Ranger KMS PID: 4247
I20260620 08:15:42.728384 25574 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.249.148:60429
Jun 20, 2026 8:15:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:15:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:15:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:15:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:15:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:15:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 08:15:45.014302 25574 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.672s user 0.008s sys 0.349s
I20260620 08:15:45.014417 25574 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 08:15:45.014498 25574 mini_ranger_kms.cc:401] 127.24.249.148:60429/kms/v1/keys
Jun 20 08:15:45 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781943345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.249.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 08:15:45 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.249.148@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
Jun 20 08:15:46 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.249.148@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
Jun 20 08:15:48 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:15:48.505236 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42477
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:42477
--ranger_config_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:60429/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:15:48.651376 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:48.651640 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:48.651739 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:48.653023 4297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:15:48.653066 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:48.653080 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:48.653093 4297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:15:48.653142 4297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:15:48.655187 4297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.249.148:60429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:42477
--ranger_config_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:48.655504 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:48.655766 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:48.661671 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:48.661827 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:48.661922 4297 server_base.cc:1034] running on GCE node
W20260620 08:15:48.661847 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:48.662273 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:48.662595 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:48.663791 4297 hybrid_clock.cc:648] HybridClock initialized: now 1781943348663776 us; error 29 us; skew 500 ppm
Jun 20 08:15:48 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943348, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:48.666512 4297 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:15:48.667073 4297 webserver.cc:460] Webserver started at http://127.24.249.190:40457/ using document root <none> and password file <none>
I20260620 08:15:48.667311 4297 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:48.667351 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:48.667491 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:15:48 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943348, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:15:48.715080 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "d631ebe6ff824a9da0737ed9ba782608"
format_stamp: "Formatted at 2026-06-20 08:15:48 on dist-test-slave-65w8"
server_key: "cb3c70137ae5a48dd9255e71335a6daf"
server_key_iv: "6e45dcc29f03058723470a663815876f"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:48.715492 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "d631ebe6ff824a9da0737ed9ba782608"
format_stamp: "Formatted at 2026-06-20 08:15:48 on dist-test-slave-65w8"
server_key: "cb3c70137ae5a48dd9255e71335a6daf"
server_key_iv: "6e45dcc29f03058723470a663815876f"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:48.717036 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:15:48.728770 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:48.729004 4297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:15:48.729130 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "d631ebe6ff824a9da0737ed9ba782608"
format_stamp: "Formatted at 2026-06-20 08:15:48 on dist-test-slave-65w8"
server_key: "cb3c70137ae5a48dd9255e71335a6daf"
server_key_iv: "6e45dcc29f03058723470a663815876f"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:48.729250 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:48.758311 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:48.760429 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:48.760725 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:48.769276 4297 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:42477
I20260620 08:15:48.769300 4364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:42477 every 8 connection(s)
I20260620 08:15:48.769907 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:15:48.770780 4365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:48.773581 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 4297
I20260620 08:15:48.773716 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:15:48.773911 4365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608: Bootstrap starting.
I20260620 08:15:48.773950 25574 external_mini_cluster.cc:1428] Setting key e1165a3950cf8ea7f30f745b19704785
I20260620 08:15:48.774608 4365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:48.774926 4365 log.cc:824] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:48.775740 4365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608: No bootstrap required, opened a new log
I20260620 08:15:48.778700 4365 raft_consensus.cc:348] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631ebe6ff824a9da0737ed9ba782608" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42477 } }
I20260620 08:15:48.778863 4365 raft_consensus.cc:374] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:48.778925 4365 raft_consensus.cc:729] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d631ebe6ff824a9da0737ed9ba782608, State: Initialized, Role: FOLLOWER
I20260620 08:15:48.779064 4365 consensus_queue.cc:260] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631ebe6ff824a9da0737ed9ba782608" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42477 } }
I20260620 08:15:48.779152 4365 raft_consensus.cc:388] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:15:48.779186 4365 raft_consensus.cc:482] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:15:48.779250 4365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:15:48.780148 4365 raft_consensus.cc:504] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631ebe6ff824a9da0737ed9ba782608" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42477 } }
I20260620 08:15:48.780264 4365 leader_election.cc:302] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d631ebe6ff824a9da0737ed9ba782608; no voters:
I20260620 08:15:48.780467 4365 leader_election.cc:288] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:15:48.780527 4370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:15:48.780772 4370 raft_consensus.cc:686] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [term 1 LEADER]: Becoming Leader. State: Replica: d631ebe6ff824a9da0737ed9ba782608, State: Running, Role: LEADER
I20260620 08:15:48.780884 4365 sys_catalog.cc:564] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:15:48.780915 4370 consensus_queue.cc:237] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631ebe6ff824a9da0737ed9ba782608" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42477 } }
I20260620 08:15:48.781307 4370 sys_catalog.cc:455] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d631ebe6ff824a9da0737ed9ba782608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631ebe6ff824a9da0737ed9ba782608" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42477 } } }
I20260620 08:15:48.781369 4370 sys_catalog.cc:458] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [sys.catalog]: This master's current role is: LEADER
I20260620 08:15:48.781482 4370 sys_catalog.cc:455] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d631ebe6ff824a9da0737ed9ba782608. Latest consensus state: current_term: 1 leader_uuid: "d631ebe6ff824a9da0737ed9ba782608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631ebe6ff824a9da0737ed9ba782608" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42477 } } }
I20260620 08:15:48.781520 4370 sys_catalog.cc:458] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608 [sys.catalog]: This master's current role is: LEADER
I20260620 08:15:48.782480 4374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 08:15:48 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:48.782927 4365 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:15:48.783056 4374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:15:48.788885 4374 catalog_manager.cc:1347] Generated new cluster ID: 615e9253f2f2438db1c083b629af2f4f
I20260620 08:15:48.789098 4374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:15:48.793901 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:48.774520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.24.249.190:42477)
0620 08:15:48.774909 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:48.774916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:48.774941 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:15:48.775510 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:48.775519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:48.775581 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:48.775729 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:48.776523 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:48.778063 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:48.779090 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:48.779746 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:48.779887 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:48.783227 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:48.783231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:48.783232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:48.783245 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:48.789778 (+ 6533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:48.790716 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:48.790720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:48.790720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:48.790867 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:48.791090 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:48.791091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:48.791091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:48.791705 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:48.792502 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:48.792598 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 08:15:48.793292 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":73,"threads_started":1}
I20260620 08:15:48.802098 4374 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:15:48.803054 4374 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:15:48.813604 4374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608: Generated new TSK 0
I20260620 08:15:48.813803 4374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:15:49.597376 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:60429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42477
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:15:49.722584 4410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:49.722832 4410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:49.722874 4410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:49.724136 4410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:49.724211 4410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:49.724258 4410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:15:49.725773 4410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.249.148:60429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:49.726065 4410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:49.726316 4410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:49.731765 4415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:49.731783 4416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:49.732054 4418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:49.732092 4410 server_base.cc:1034] running on GCE node
I20260620 08:15:49.732307 4410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:49.732582 4410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:49.733826 4410 hybrid_clock.cc:648] HybridClock initialized: now 1781943349733791 us; error 46 us; skew 500 ppm
Jun 20 08:15:49 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943349, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:49.737567 4410 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:15:49.738162 4410 webserver.cc:460] Webserver started at http://127.24.249.129:37911/ using document root <none> and password file <none>
I20260620 08:15:49.738428 4410 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:49.738495 4410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:49.738631 4410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:15:49 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943349, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:15:49.754274 4410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "52c2f18e27ee466c8799f49456d09fe9"
format_stamp: "Formatted at 2026-06-20 08:15:49 on dist-test-slave-65w8"
server_key: "a9ed7ed7b5af7e17beaf75bdf20e4561"
server_key_iv: "c79a35fa70dbe7274dfec0d2aea4444e"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:49.754841 4410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "52c2f18e27ee466c8799f49456d09fe9"
format_stamp: "Formatted at 2026-06-20 08:15:49 on dist-test-slave-65w8"
server_key: "a9ed7ed7b5af7e17beaf75bdf20e4561"
server_key_iv: "c79a35fa70dbe7274dfec0d2aea4444e"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:49.756880 4410 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:15:49.773924 4425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:49.774259 4410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:15:49.774379 4410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "52c2f18e27ee466c8799f49456d09fe9"
format_stamp: "Formatted at 2026-06-20 08:15:49 on dist-test-slave-65w8"
server_key: "a9ed7ed7b5af7e17beaf75bdf20e4561"
server_key_iv: "c79a35fa70dbe7274dfec0d2aea4444e"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:49.774463 4410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:49.797117 4410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:49.799757 4410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:49.799923 4410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:49.800143 4410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:15:49.800522 4410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:15:49.800570 4410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:49.800607 4410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:15:49.800627 4410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:49.809737 4410 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:35415
I20260620 08:15:49.810223 4410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:15:49.814666 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 4410
I20260620 08:15:49.814780 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:15:49.815032 25574 external_mini_cluster.cc:1428] Setting key 83c754fd9f85543d94855f97d8246f4b
I20260620 08:15:49.815377 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:35415 every 8 connection(s)
Jun 20 08:15:49 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943349, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:49.829674 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:49.816001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:41653 (local address 127.24.249.190:42477)
0620 08:15:49.817137 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:49.817140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:49.817432 (+ 292us) server_negotiation.cc:407] Connection header received
0620 08:15:49.818333 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:49.818335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:49.818350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:49.818390 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:49.819462 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:49.820275 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:49.822252 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:49.822556 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:49.822600 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:49.825919 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:49.825921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:49.825921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:49.825930 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:49.828027 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:49.828530 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:49.828532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:49.828532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:49.828593 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:49.828854 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:49.828855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:49.828855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:49.828991 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:49.829051 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:49.829463 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 08:15:49.829566 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":88,"threads_started":1}
I20260620 08:15:49.829825 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:49.816009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42477 (local address 127.24.249.129:41653)
0620 08:15:49.817263 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:49.817317 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:49.818196 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:49.818596 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:49.818601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:49.818971 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:15:49.819348 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:49.819355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:49.820394 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:49.820395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:49.822138 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:49.822141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:49.822693 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:49.822694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:49.822917 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:49.823723 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:15:49.823734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:15:49.825777 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:15:49.828134 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:49.828142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:49.828145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:49.828443 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:49.828660 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:49.828660 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:49.828661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:49.828788 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:49.829083 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:15:49.829090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:15:49.829254 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 08:15:49.829359 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:15:49.829513 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"thread_start_us":83,"threads_started":1}
I20260620 08:15:49.830394 4541 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42477
I20260620 08:15:49.830515 4541 heartbeater.cc:464] Registering TS with master...
I20260620 08:15:49.830727 4541 heartbeater.cc:511] Master 127.24.249.190:42477 requested a full tablet report, sending...
I20260620 08:15:49.831305 4329 ts_manager.cc:194] Registered new tserver with Master: 52c2f18e27ee466c8799f49456d09fe9 (127.24.249.129:35415)
I20260620 08:15:49.832242 4329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:41653
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:15:49.861095 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:60429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42477
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:15:49 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781943349, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 08:15:50.007025 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:50.007319 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:50.007357 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:50.009243 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:50.009320 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:50.009363 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:15:50.011847 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.249.148:60429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:50.012122 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:50.012476 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:50.023319 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:50.023576 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:50.024745 4547 server_base.cc:1034] running on GCE node
W20260620 08:15:50.024921 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:50.025247 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:50.025537 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:50.026737 4547 hybrid_clock.cc:648] HybridClock initialized: now 1781943350026695 us; error 54 us; skew 500 ppm
Jun 20 08:15:50 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943350, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:50.031702 4547 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:15:50.032335 4547 webserver.cc:460] Webserver started at http://127.24.249.130:39515/ using document root <none> and password file <none>
I20260620 08:15:50.032665 4547 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:50.032783 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:50.032965 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:15:50 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943350, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:15:50.055704 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "560d36e0f1894fb6ab7a00325962639f"
format_stamp: "Formatted at 2026-06-20 08:15:50 on dist-test-slave-65w8"
server_key: "1407a936121cf29c211f2d07c5f1aa9e"
server_key_iv: "2b9f3be276c39a1313eb75fd31a52d4f"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:50.058315 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "560d36e0f1894fb6ab7a00325962639f"
format_stamp: "Formatted at 2026-06-20 08:15:50 on dist-test-slave-65w8"
server_key: "1407a936121cf29c211f2d07c5f1aa9e"
server_key_iv: "2b9f3be276c39a1313eb75fd31a52d4f"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:50.061605 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 08:15:50.081148 4564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:50.081432 4547 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260620 08:15:50.081537 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "560d36e0f1894fb6ab7a00325962639f"
format_stamp: "Formatted at 2026-06-20 08:15:50 on dist-test-slave-65w8"
server_key: "1407a936121cf29c211f2d07c5f1aa9e"
server_key_iv: "2b9f3be276c39a1313eb75fd31a52d4f"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:50.081640 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:50.098223 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:50.100629 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:50.100808 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:50.101127 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:15:50.101567 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:15:50.101624 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:50.101665 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:15:50.101691 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:50.113698 4547 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:41329
I20260620 08:15:50.118129 4678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:41329 every 8 connection(s)
I20260620 08:15:50.118649 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:15:50.119407 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 4547
I20260620 08:15:50.119513 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:15:50.119731 25574 external_mini_cluster.cc:1428] Setting key 3e2d831c3836d8b60b35072defdb80b4
Jun 20 08:15:50 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943350, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:50.141031 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.120445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:59539 (local address 127.24.249.190:42477)
0620 08:15:50.120511 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.120514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.125434 (+ 4920us) server_negotiation.cc:407] Connection header received
0620 08:15:50.126266 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.126268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.126288 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.126322 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:50.127358 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.128175 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.133971 (+ 5796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.134306 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.134355 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.137517 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:50.137519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:50.137519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:50.137528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:50.139269 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:50.139802 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:50.139803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:50.139804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:50.139865 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:50.140158 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:50.140159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:50.140159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:50.140307 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:50.140351 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.140810 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.140917 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:15:50.142036 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.120220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42477 (local address 127.24.249.130:59539)
0620 08:15:50.125269 (+ 5049us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.125332 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:50.126054 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:50.126524 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:50.126528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:50.126893 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:15:50.127226 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.127233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.128297 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.128298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.129586 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.129589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.134467 (+ 4878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.134468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.134663 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.135370 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:15:50.135382 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:15:50.137394 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:15:50.139377 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:50.139386 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:50.139388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:50.139703 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:50.139938 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:50.139939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:50.139939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:50.140073 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:50.140399 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:15:50.140405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:15:50.140574 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 08:15:50.141417 (+ 843us) client_negotiation.cc:240] Negotiation successful
0620 08:15:50.141597 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4989,"thread_start_us":88,"threads_started":1}
I20260620 08:15:50.142815 4679 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42477
I20260620 08:15:50.142922 4679 heartbeater.cc:464] Registering TS with master...
I20260620 08:15:50.143131 4679 heartbeater.cc:511] Master 127.24.249.190:42477 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:15:50.143537 4329 ts_manager.cc:194] Registered new tserver with Master: 560d36e0f1894fb6ab7a00325962639f (127.24.249.130:41329)
I20260620 08:15:50.144019 4329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:59539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:15:50.165086 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:60429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42477
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:15:50.311088 4684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:15:50.311501 4684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:15:50.311545 4684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:15:50.313666 4684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:15:50.313750 4684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:15:50.313797 4684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:15:50.316488 4684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.249.148:60429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:15:50.316917 4684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:15:50.317225 4684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:15:50.325450 4689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:15:50.325736 4692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:50.326613 4684 server_base.cc:1034] running on GCE node
W20260620 08:15:50.326867 4690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:15:50.327139 4684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:15:50.327418 4684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:15:50.328577 4684 hybrid_clock.cc:648] HybridClock initialized: now 1781943350328553 us; error 35 us; skew 500 ppm
Jun 20 08:15:50 dist-test-slave-65w8 krb5kdc[3853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943350, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:15:50.332875 4684 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:15:50.333444 4684 webserver.cc:460] Webserver started at http://127.24.249.131:41981/ using document root <none> and password file <none>
I20260620 08:15:50.333765 4684 fs_manager.cc:359] Metadata directory not provided
I20260620 08:15:50.333907 4684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:15:50.334087 4684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:15:50 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943350, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:15:50.351226 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "4c9c58b0202d42eaaa9e732908dfd585"
format_stamp: "Formatted at 2026-06-20 08:15:50 on dist-test-slave-65w8"
server_key: "f02efe33819cb96ce28d1961e3c425ad"
server_key_iv: "dbffc1704b7d8a9dcadffff052486c7a"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:50.352058 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "4c9c58b0202d42eaaa9e732908dfd585"
format_stamp: "Formatted at 2026-06-20 08:15:50 on dist-test-slave-65w8"
server_key: "f02efe33819cb96ce28d1961e3c425ad"
server_key_iv: "dbffc1704b7d8a9dcadffff052486c7a"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:50.357138 4684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 08:15:50.377787 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:50.378008 4684 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260620 08:15:50.378105 4684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "4c9c58b0202d42eaaa9e732908dfd585"
format_stamp: "Formatted at 2026-06-20 08:15:50 on dist-test-slave-65w8"
server_key: "f02efe33819cb96ce28d1961e3c425ad"
server_key_iv: "dbffc1704b7d8a9dcadffff052486c7a"
server_key_version: "kuduclusterkey@0"
I20260620 08:15:50.378216 4684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:15:50.403106 4684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:15:50.406258 4684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:15:50.406484 4684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:15:50.406818 4684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:15:50.407289 4684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:15:50.407377 4684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:50.407444 4684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:15:50.407498 4684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:15:50.416605 4684 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:40675
I20260620 08:15:50.416720 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:40675 every 8 connection(s)
I20260620 08:15:50.417225 4684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:15:50.424219 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 4684
I20260620 08:15:50.424345 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:15:50.424526 25574 external_mini_cluster.cc:1428] Setting key da04d419abb69346c8a7334bc9ee0f87
Jun 20 08:15:50 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943350, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:15:50.431596 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.418431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:55957 (local address 127.24.249.190:42477)
0620 08:15:50.418482 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.418485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.418790 (+ 305us) server_negotiation.cc:407] Connection header received
0620 08:15:50.419428 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.419430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.419446 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.419531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:50.420248 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.421284 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.422764 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.423235 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.423352 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.427350 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:50.427352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:50.427352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:50.427371 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:50.429775 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:50.430320 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:50.430322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:50.430323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:50.430425 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:50.430772 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:50.430773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:50.430774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:50.430937 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:50.431019 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.431364 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.431465 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:15:50.431905 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.418415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42477 (local address 127.24.249.131:55957)
0620 08:15:50.418633 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.418682 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:50.419302 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:50.419604 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:50.419609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:50.419853 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:15:50.420127 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.420141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.421416 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.421417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.422638 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.422641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.423362 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.423363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.423524 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.424822 (+ 1298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:15:50.424832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:15:50.427220 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:15:50.429896 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:50.429905 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:50.429908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:50.430216 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:50.430524 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:15:50.430525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:15:50.430525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:15:50.430682 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:15:50.431015 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:15:50.431021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:15:50.431191 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 08:15:50.431300 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 08:15:50.431463 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260620 08:15:50.432356 4813 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42477
I20260620 08:15:50.432503 4813 heartbeater.cc:464] Registering TS with master...
I20260620 08:15:50.432857 4813 heartbeater.cc:511] Master 127.24.249.190:42477 requested a full tablet report, sending...
I20260620 08:15:50.433398 4329 ts_manager.cc:194] Registered new tserver with Master: 4c9c58b0202d42eaaa9e732908dfd585 (127.24.249.131:40675)
I20260620 08:15:50.433931 4329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:55957
I20260620 08:15:50.442898 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:15:50.453066 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.444732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.24.249.190:42477)
0620 08:15:50.444778 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.444780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.445224 (+ 444us) server_negotiation.cc:407] Connection header received
0620 08:15:50.445319 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.445320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.445330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.445358 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:15:50.446270 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.447030 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.448182 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.448471 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.448511 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.450195 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:15:50.450196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:15:50.450196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:15:50.450204 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:15:50.451852 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:50.452267 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:50.452268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:50.452269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:50.452323 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:15:50.452506 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:15:50.452507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:15:50.452507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:15:50.452621 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:15:50.452703 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.452905 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.452976 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:15:50.455355 4329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:15:50.457701 4329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:15:50.482820 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.467429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:40675 (local address 127.0.0.1:52468)
0620 08:15:50.472902 (+ 5473us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.472914 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:50.472976 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:50.473814 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:50.473815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:50.473816 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:50.473846 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.473847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.476756 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.476757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.479587 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.479590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.481914 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.481915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.482560 (+ 645us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.482591 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 08:15:50.482662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:15:50.482721 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5453,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260620 08:15:50.483177 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.467707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.24.249.131:40675)
0620 08:15:50.468806 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.468810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.473056 (+ 4246us) server_negotiation.cc:407] Connection header received
0620 08:15:50.473109 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.473112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.473130 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.473206 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:50.475617 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.476655 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.479694 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.480835 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.480979 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.481060 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.483043 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.483092 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":108,"threads_started":1}
I20260620 08:15:50.483909 4747 tablet_service.cc:1467] Processing CreateTablet for tablet ffb0ebeb60a2474d99090c0032a4fd30 (DEFAULT_TABLE table=test-table [id=8167cde92d9043b580ca567927c5aaec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:15:50.484333 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb0ebeb60a2474d99090c0032a4fd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:50.484349 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.467913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.24.249.130:41329)
0620 08:15:50.468726 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.468731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.469060 (+ 329us) server_negotiation.cc:407] Connection header received
0620 08:15:50.469103 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.469105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.469122 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.469187 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:50.469413 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.480511 (+ 11098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.482107 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.482844 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.483812 (+ 968us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.483896 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.484211 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.484261 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":101,"threads_started":1}
I20260620 08:15:50.484663 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.467429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:41329 (local address 127.0.0.1:46370)
0620 08:15:50.468868 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.468912 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:50.468983 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:50.469270 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:50.469273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:50.469275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:50.469343 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.469345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.480637 (+ 11292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.480639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.481976 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.481980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.483934 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.483936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.483998 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.484002 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:15:50.484050 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:15:50.484087 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409,"thread_start_us":79,"threads_started":1}
I20260620 08:15:50.485474 4613 tablet_service.cc:1467] Processing CreateTablet for tablet ffb0ebeb60a2474d99090c0032a4fd30 (DEFAULT_TABLE table=test-table [id=8167cde92d9043b580ca567927c5aaec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:15:50.485819 4613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb0ebeb60a2474d99090c0032a4fd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:50.488811 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.467674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:35415 (local address 127.0.0.1:39636)
0620 08:15:50.473315 (+ 5641us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.473325 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:50.473367 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:50.473683 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:50.473684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:50.473686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:50.473737 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.473739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.475207 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.475208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.478962 (+ 3754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.478965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.488545 (+ 9580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.488547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.488623 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.488628 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:15:50.488687 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 08:15:50.488738 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5622,"thread_start_us":144,"threads_started":1}
I20260620 08:15:50.489204 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.467949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.24.249.129:35415)
0620 08:15:50.473477 (+ 5528us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.473482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.473492 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:15:50.473524 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.473526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.473542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.473610 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:50.473948 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.475106 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.481452 (+ 6346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.485526 (+ 4074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.485602 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.487078 (+ 1476us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.489060 (+ 1982us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.489121 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5490,"thread_start_us":61,"threads_started":1}
I20260620 08:15:50.490144 4475 tablet_service.cc:1467] Processing CreateTablet for tablet ffb0ebeb60a2474d99090c0032a4fd30 (DEFAULT_TABLE table=test-table [id=8167cde92d9043b580ca567927c5aaec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:15:50.490476 4475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb0ebeb60a2474d99090c0032a4fd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:15:50.495076 4833 tablet_bootstrap.cc:492] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Bootstrap starting.
I20260620 08:15:50.495316 4834 tablet_bootstrap.cc:492] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9: Bootstrap starting.
I20260620 08:15:50.496368 4834 tablet_bootstrap.cc:654] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:50.497668 4833 tablet_bootstrap.cc:654] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:50.497932 4832 tablet_bootstrap.cc:492] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Bootstrap starting.
I20260620 08:15:50.498562 4834 log.cc:824] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:50.500475 4833 log.cc:824] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:50.501099 4832 tablet_bootstrap.cc:654] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Neither blocks nor log segments found. Creating new log.
I20260620 08:15:50.501489 4832 log.cc:824] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Log is configured to *not* fsync() on all Append() calls
I20260620 08:15:50.502625 4834 tablet_bootstrap.cc:492] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9: No bootstrap required, opened a new log
I20260620 08:15:50.502745 4834 ts_tablet_manager.cc:1397] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 08:15:50.505774 4834 raft_consensus.cc:348] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.506145 4834 raft_consensus.cc:374] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:50.506246 4834 raft_consensus.cc:729] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c2f18e27ee466c8799f49456d09fe9, State: Initialized, Role: FOLLOWER
I20260620 08:15:50.506465 4834 consensus_queue.cc:260] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.506875 4834 ts_tablet_manager.cc:1428] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:15:50.507087 4541 heartbeater.cc:503] Master 127.24.249.190:42477 was elected leader, sending a full tablet report...
I20260620 08:15:50.508884 4833 tablet_bootstrap.cc:492] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: No bootstrap required, opened a new log
I20260620 08:15:50.509033 4833 ts_tablet_manager.cc:1397] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260620 08:15:50.511334 4833 raft_consensus.cc:348] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.511575 4833 raft_consensus.cc:374] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:50.511662 4833 raft_consensus.cc:729] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560d36e0f1894fb6ab7a00325962639f, State: Initialized, Role: FOLLOWER
I20260620 08:15:50.511862 4833 consensus_queue.cc:260] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.512305 4833 ts_tablet_manager.cc:1428] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:15:50.512467 4679 heartbeater.cc:503] Master 127.24.249.190:42477 was elected leader, sending a full tablet report...
I20260620 08:15:50.513928 4832 tablet_bootstrap.cc:492] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: No bootstrap required, opened a new log
I20260620 08:15:50.514071 4832 ts_tablet_manager.cc:1397] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260620 08:15:50.516413 4832 raft_consensus.cc:348] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.516660 4832 raft_consensus.cc:374] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:15:50.516750 4832 raft_consensus.cc:729] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9c58b0202d42eaaa9e732908dfd585, State: Initialized, Role: FOLLOWER
I20260620 08:15:50.516942 4832 consensus_queue.cc:260] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.517306 4832 ts_tablet_manager.cc:1428] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:15:50.517475 4813 heartbeater.cc:503] Master 127.24.249.190:42477 was elected leader, sending a full tablet report...
I20260620 08:15:50.558370 4838 raft_consensus.cc:482] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:15:50.558530 4838 raft_consensus.cc:504] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.558913 4838 leader_election.cc:288] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 560d36e0f1894fb6ab7a00325962639f (127.24.249.130:41329), 4c9c58b0202d42eaaa9e732908dfd585 (127.24.249.131:40675)
I20260620 08:15:50.563946 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.559758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:58493 (local address 127.24.249.130:41329)
0620 08:15:50.559797 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.559799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.560152 (+ 353us) server_negotiation.cc:407] Connection header received
0620 08:15:50.560264 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.560266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.560277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.560308 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:50.560556 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.561327 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.562919 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.563518 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.563572 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.563582 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.563806 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.563849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:15:50.564940 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.559206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:41329 (local address 127.24.249.129:58493)
0620 08:15:50.560078 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.560091 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:50.560183 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:50.560379 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:50.560379 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:50.560392 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:50.560473 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.560476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.561454 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.561455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.562805 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.562808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.563645 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.563646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.563722 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.563726 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:15:50.564778 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0620 08:15:50.564829 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":121,"threads_started":1}
I20260620 08:15:50.565515 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb0ebeb60a2474d99090c0032a4fd30" candidate_uuid: "52c2f18e27ee466c8799f49456d09fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560d36e0f1894fb6ab7a00325962639f" is_pre_election: true
I20260620 08:15:50.565766 4620 raft_consensus.cc:2454] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c2f18e27ee466c8799f49456d09fe9 in term 0.
I20260620 08:15:50.566051 4426 leader_election.cc:302] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c2f18e27ee466c8799f49456d09fe9, 560d36e0f1894fb6ab7a00325962639f; no voters:
I20260620 08:15:50.566205 4838 raft_consensus.cc:2781] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:15:50.566255 4838 raft_consensus.cc:482] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:15:50.566284 4838 raft_consensus.cc:3037] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 0 FOLLOWER]: Advancing to term 1
W20260620 08:15:50.566764 4542 tablet.cc:2369] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:15:50.567304 4838 raft_consensus.cc:504] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.567451 4838 leader_election.cc:288] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [CANDIDATE]: Term 1 election: Requested vote from peers 560d36e0f1894fb6ab7a00325962639f (127.24.249.130:41329), 4c9c58b0202d42eaaa9e732908dfd585 (127.24.249.131:40675)
I20260620 08:15:50.567659 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb0ebeb60a2474d99090c0032a4fd30" candidate_uuid: "52c2f18e27ee466c8799f49456d09fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560d36e0f1894fb6ab7a00325962639f"
I20260620 08:15:50.567749 4620 raft_consensus.cc:3037] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:15:50.568634 4620 raft_consensus.cc:2454] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c2f18e27ee466c8799f49456d09fe9 in term 1.
I20260620 08:15:50.568813 4426 leader_election.cc:302] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c2f18e27ee466c8799f49456d09fe9, 560d36e0f1894fb6ab7a00325962639f; no voters:
I20260620 08:15:50.568898 4838 raft_consensus.cc:2781] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:15:50.569042 4838 raft_consensus.cc:686] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 52c2f18e27ee466c8799f49456d09fe9, State: Running, Role: LEADER
I20260620 08:15:50.569146 4838 consensus_queue.cc:237] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } }
I20260620 08:15:50.569954 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.559543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:54021 (local address 127.24.249.131:40675)
0620 08:15:50.559591 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.559594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.564465 (+ 4871us) server_negotiation.cc:407] Connection header received
0620 08:15:50.564588 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.564590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.564600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.564630 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:15:50.564913 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.565634 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.568148 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.568754 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.568808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.568819 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.569810 (+ 991us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.569881 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:15:50.570003 4329 catalog_manager.cc:5495] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c2f18e27ee466c8799f49456d09fe9 (127.24.249.129). New cstate: current_term: 1 leader_uuid: "52c2f18e27ee466c8799f49456d09fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c2f18e27ee466c8799f49456d09fe9" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 35415 } health_report { overall_health: HEALTHY } } }
I20260620 08:15:50.570219 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.559466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:40675 (local address 127.24.249.129:54021)
0620 08:15:50.564396 (+ 4930us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.564408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:15:50.564498 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:15:50.564700 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:15:50.564701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:15:50.564703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:15:50.564829 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.564830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.566929 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.566930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.568038 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:15:50.568041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.569643 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.569644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:15:50.569720 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.569724 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:15:50.570081 (+ 357us) client_negotiation.cc:240] Negotiation successful
0620 08:15:50.570128 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4871,"thread_start_us":79,"threads_started":1}
I20260620 08:15:50.570683 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb0ebeb60a2474d99090c0032a4fd30" candidate_uuid: "52c2f18e27ee466c8799f49456d09fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9c58b0202d42eaaa9e732908dfd585" is_pre_election: true
I20260620 08:15:50.570827 4767 raft_consensus.cc:2454] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c2f18e27ee466c8799f49456d09fe9 in term 0.
I20260620 08:15:50.570958 4766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb0ebeb60a2474d99090c0032a4fd30" candidate_uuid: "52c2f18e27ee466c8799f49456d09fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c9c58b0202d42eaaa9e732908dfd585"
I20260620 08:15:50.571020 4766 raft_consensus.cc:3037] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:15:50.571858 4766 raft_consensus.cc:2454] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c2f18e27ee466c8799f49456d09fe9 in term 1.
I20260620 08:15:50.593806 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.589890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.24.249.129:35415)
0620 08:15:50.589956 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:15:50.589958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:15:50.589975 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:15:50.590020 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:15:50.590021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:15:50.590046 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:15:50.590121 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:15:50.591086 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.591832 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.592637 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:15:50.592909 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:15:50.592972 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:15:50.593186 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:15:50.593323 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:15:50.593403 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:15:50.593677 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 08:15:50.593704 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:15:50.603518 4766 raft_consensus.cc:1261] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 1 FOLLOWER]: Refusing update from remote peer 52c2f18e27ee466c8799f49456d09fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:15:50.603622 4620 raft_consensus.cc:1261] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 1 FOLLOWER]: Refusing update from remote peer 52c2f18e27ee466c8799f49456d09fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:15:50.604025 4843 consensus_queue.cc:1035] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560d36e0f1894fb6ab7a00325962639f" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:15:50.604152 4838 consensus_queue.cc:1035] T ffb0ebeb60a2474d99090c0032a4fd30 P 52c2f18e27ee466c8799f49456d09fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c9c58b0202d42eaaa9e732908dfd585" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 40675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:15:50.612550 4847 mvcc.cc:205] Tried to move back new op lower bound from 7298839964068421632 to 7298839963931729920. Current Snapshot: MvccSnapshot[applied={T|T < 7298839964068421632}]
I20260620 08:15:50.614339 4848 mvcc.cc:205] Tried to move back new op lower bound from 7298839964068421632 to 7298839963931729920. Current Snapshot: MvccSnapshot[applied={T|T < 7298839964068421632}]
I20260620 08:15:50.624575 4329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:15:50.624701 4329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:15:50.625487 4329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d631ebe6ff824a9da0737ed9ba782608: Sending DeleteTablet for 3 replicas of tablet ffb0ebeb60a2474d99090c0032a4fd30
I20260620 08:15:50.626015 4613 tablet_service.cc:1514] Processing DeleteTablet for tablet ffb0ebeb60a2474d99090c0032a4fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:15:50 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:46370
I20260620 08:15:50.626293 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet ffb0ebeb60a2474d99090c0032a4fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:15:50 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:52468
I20260620 08:15:50.626674 4475 tablet_service.cc:1514] Processing DeleteTablet for tablet ffb0ebeb60a2474d99090c0032a4fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:15:50 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:39636
I20260620 08:15:50.627123 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 4410
I20260620 08:15:50.628495 4855 tablet_replica.cc:330] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: stopping tablet replica
I20260620 08:15:50.628639 4855 raft_consensus.cc:2229] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:15:50.628762 4855 raft_consensus.cc:2258] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:15:50.628968 4856 tablet_replica.cc:330] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: stopping tablet replica
Jun 20 08:15:50 dist-test-slave-65w8 krb5kdc[3853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943349, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:15:50.629070 4856 raft_consensus.cc:2229] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:15:50.629166 4856 raft_consensus.cc:2258] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:15:50.632612 4855 ts_tablet_manager.cc:1905] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:15:50.632627 4856 ts_tablet_manager.cc:1905] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:15:50.634349 4855 ts_tablet_manager.cc:1918] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:15:50.634428 4855 log.cc:1196] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/ffb0ebeb60a2474d99090c0032a4fd30
I20260620 08:15:50.634419 4856 ts_tablet_manager.cc:1918] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:15:50.634486 4856 log.cc:1196] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/ffb0ebeb60a2474d99090c0032a4fd30
I20260620 08:15:50.634683 4855 ts_tablet_manager.cc:1939] T ffb0ebeb60a2474d99090c0032a4fd30 P 560d36e0f1894fb6ab7a00325962639f: Deleting consensus metadata
I20260620 08:15:50.635107 4856 ts_tablet_manager.cc:1939] T ffb0ebeb60a2474d99090c0032a4fd30 P 4c9c58b0202d42eaaa9e732908dfd585: Deleting consensus metadata
I20260620 08:15:50.635108 4313 catalog_manager.cc:4841] TS 560d36e0f1894fb6ab7a00325962639f (127.24.249.130:41329): tablet ffb0ebeb60a2474d99090c0032a4fd30 (table test-table [id=8167cde92d9043b580ca567927c5aaec]) successfully deleted
I20260620 08:15:50.635562 4315 catalog_manager.cc:4841] TS 4c9c58b0202d42eaaa9e732908dfd585 (127.24.249.131:40675): tablet ffb0ebeb60a2474d99090c0032a4fd30 (table test-table [id=8167cde92d9043b580ca567927c5aaec]) successfully deleted
W20260620 08:15:50.644745 4314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:35415 (error 108)
I20260620 08:15:50.645179 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 4547
I20260620 08:15:50.653841 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 4684
I20260620 08:15:50.655169 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:15:50.655019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:35415 (local address 127.0.0.1:39664)
0620 08:15:50.655084 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 08:15:50.655124 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:15:50.655303 4314 catalog_manager.cc:4584] TS 52c2f18e27ee466c8799f49456d09fe9 (127.24.249.129:35415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffb0ebeb60a2474d99090c0032a4fd30: Network error: Client connection negotiation failed: client connection to 127.24.249.129:35415: connect: Connection refused (error 111)
I20260620 08:15:50.660113 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 4297
I20260620 08:15:50.667477 25574 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 08:15:51.012319 25574 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 08:15:51.012454 25574 mini_ranger.cc:66] Stopping Ranger...
I20260620 08:15:51.357724 25574 mini_ranger.cc:68] Stopped Ranger
2026-06-20 08:15:51.357 UTC [3883] LOG: received smart shutdown request
2026-06-20 08:15:51.361 UTC [3883] LOG: background worker "logical replication launcher" (PID 3892) exited with exit code 1
2026-06-20 08:15:51.362 UTC [3887] LOG: shutting down
2026-06-20 08:15:51.565 UTC [3883] LOG: database system is shut down
2026-06-20T08:15:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50258 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:15:51 dist-test-slave-65w8 krb5kdc[4867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:15:51 dist-test-slave-65w8 krb5kdc[4867](info): set up 2 sockets
Jun 20 08:15:51 dist-test-slave-65w8 krb5kdc[4867](info): commencing operation
krb5kdc: starting...
W20260620 08:15:52.742671 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:15:52 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:15:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/rangeradmin_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/rangeradmin_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/rangerlookup_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/rangerlookup_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
I20260620 08:15:52.930555 25574 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5EkY3n/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/postgres -l logfile start
2026-06-20 08:15:55.668 UTC [4897] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 08:15:55.668 UTC [4897] LOG: listening on IPv4 address "127.24.249.148", port 43329
2026-06-20 08:15:55.672 UTC [4897] LOG: listening on Unix socket "/tmp/.s.PGSQL.43329"
2026-06-20 08:15:55.693 UTC [4900] LOG: database system was shut down at 2026-06-20 08:15:54 UTC
2026-06-20 08:15:55.697 UTC [4897] LOG: database system is ready to accept connections
I20260620 08:15:56.681967 25574 mini_postgres.cc:96] Postgres bound to 43329
2026-06-20 08:15:56.688 UTC [4908] FATAL: database "slave" does not exist
127.24.249.148:43329 - accepting connections
I20260620 08:15:56.689294 25574 mini_ranger.cc:161] Starting Ranger...
I20260620 08:15:56.705121 25574 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 08:15:57.302594 25574 mini_ranger.cc:87] Created ranger Postgres database
I20260620 08:15:57.302703 25574 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-admin
2026-06-20 08:15:57,598 [I] DB FLAVOR :POSTGRES
2026-06-20 08:15:57,598 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:15:57,599 [I] Checking connection..
2026-06-20 08:15:57,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 08:15:57,864 [I] Checking connection passed.
2026-06-20 08:15:57,864 [I] --------- Verifying version history table ---------
2026-06-20 08:15:57,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:15:58,144 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:15:58,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:15:58,383 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 08:15:58,384 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 08:15:58,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 08:15:58.641 UTC [4972] WARNING: there is no transaction in progress
2026-06-20 08:15:58,647 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 08:15:58,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 08:15:58,926 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 08:15:58,926 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:15:58,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 08:15:59,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 08:15:59,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-65w8.c.gcp-upstream.internal','N') ;"
2026-06-20 08:15:59,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:15:59,959 [I] Table x_portal_user does not exist in database ranger
2026-06-20 08:15:59,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:16:00,205 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 08:16:00,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:16:00,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:16:00,702 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 08:16:00,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 08:16:01.714 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.733 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.752 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.769 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.793 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.814 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.876 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.884 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.902 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.918 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.931 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.949 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.959 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.967 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.976 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:01.984 UTC [5080] WARNING: there is no transaction in progress
2026-06-20 08:16:02,720 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 08:16:02,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 08:16:02,955 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 08:16:02,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 08:16:03,229 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 08:16:03,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 08:16:03,501 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 08:16:03,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 08:16:03,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 08:16:04,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-65w8.c.gcp-upstream.internal';"
2026-06-20 08:16:04,254 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 08:16:04,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 08:16:04,499 [I] DB_PATCHES have already been applied
I20260620 08:16:04.506042 25574 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-admin:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5EkY3n/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 08:16:04.506139 25574 mini_ranger.cc:193] Using host: 127.24.249.148
I20260620 08:16:04.509608 25574 mini_ranger.cc:227] Ranger admin URL: http://127.24.249.148:33903
Jun 20, 2026 8:16:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:16:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:16:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:16:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:16:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:16:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 08:16:05.640976 25574 mini_ranger.cc:160] Time spent starting Ranger: real 8.952s user 0.000s sys 0.010s
Jun 20, 2026 8:16:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:16:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 8:16:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:16:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:16:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 08:16:28.708626 25574 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.249.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.249.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/rangerkms_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.249.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/rangerkms_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.249.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.249.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.249.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/HTTP_127.24.249.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 08:16:28 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943388, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 08:16:28.807106 25574 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 08:16:28.817739 25574 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 08:16:29.758558 25574 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 08:16:29.758646 25574 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms
I20260620 08:16:29.758662 25574 mini_ranger_kms.cc:227] Using postgres at 127.24.249.148:43329
2026-06-20 08:16:29,965 [I] DB FLAVOR :POSTGRES
2026-06-20 08:16:29,965 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 08:16:29,965 [I] Checking connection
2026-06-20 08:16:29,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 08:16:30,222 [I] connection success
2026-06-20 08:16:30,222 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 08:16:30,222 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 08:16:30,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 08:16:30,483 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 08:16:30,483 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 08:16:30,484 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 08:16:30,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.249.148:43329/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 08:16:30,754 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 08:16:31.241531 25574 mini_ranger_kms.cc:310] Created kms service
I20260620 08:16:31.460503 25574 mini_ranger_kms.cc:326] Created kudu user
I20260620 08:16:31.509119 25574 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 08:16:31 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943388, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:31.960078 25574 mini_ranger_kms.cc:384] Added ranger policy
I20260620 08:16:31.960341 25574 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms:/tmp/dist-test-task5EkY3n/build/release/bin/postgresql.jar:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5EkY3n/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 08:16:31.960369 25574 mini_ranger_kms.cc:240] Using host: 127.24.249.148
I20260620 08:16:31.963999 25574 mini_ranger_kms.cc:276] Ranger KMS PID: 5261
I20260620 08:16:31.964087 25574 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.249.148:40653
Jun 20, 2026 8:16:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:16:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:16:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:16:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:16:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:16:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 08:16:34.273774 25574 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.467s user 0.015s sys 0.173s
I20260620 08:16:34.273903 25574 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 08:16:34.273950 25574 mini_ranger_kms.cc:401] 127.24.249.148:40653/kms/v1/keys
Jun 20 08:16:34 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781943394, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.249.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 08:16:35 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943394, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.249.148@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
Jun 20 08:16:35 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943394, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.249.148@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
Jun 20 08:16:37 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:16:37.757472 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43591
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:43591
--ranger_config_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:37.839677 5311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:37.839893 5311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:37.839915 5311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:37.841104 5311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:37.841149 5311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:37.841163 5311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:37.841176 5311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:37.841188 5311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:37.842938 5311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.249.148:40653/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:43591
--ranger_config_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:37.843195 5311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:37.843461 5311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:37.848284 5316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:37.848412 5317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:37.848272 5319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:37.848824 5311 server_base.cc:1034] running on GCE node
I20260620 08:16:37.849057 5311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:37.849363 5311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:37.850515 5311 hybrid_clock.cc:648] HybridClock initialized: now 1781943397850491 us; error 40 us; skew 500 ppm
Jun 20 08:16:37 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943397, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:37.857733 5311 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:16:37.858261 5311 webserver.cc:460] Webserver started at http://127.24.249.190:33561/ using document root <none> and password file <none>
I20260620 08:16:37.858486 5311 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:37.858525 5311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:37.858695 5311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:16:37 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943397, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:37.906927 5311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "5b3ca23c265e4032a23ee074bb031ed5"
format_stamp: "Formatted at 2026-06-20 08:16:37 on dist-test-slave-65w8"
server_key: "c4ddbd5233744c83ec40be3c9a55b814"
server_key_iv: "3a4e1f262b86deb88ba677c774082e14"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:37.907380 5311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "5b3ca23c265e4032a23ee074bb031ed5"
format_stamp: "Formatted at 2026-06-20 08:16:37 on dist-test-slave-65w8"
server_key: "c4ddbd5233744c83ec40be3c9a55b814"
server_key_iv: "3a4e1f262b86deb88ba677c774082e14"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:37.909026 5311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:16:37.921491 5326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:37.921711 5311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:37.921814 5311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "5b3ca23c265e4032a23ee074bb031ed5"
format_stamp: "Formatted at 2026-06-20 08:16:37 on dist-test-slave-65w8"
server_key: "c4ddbd5233744c83ec40be3c9a55b814"
server_key_iv: "3a4e1f262b86deb88ba677c774082e14"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:37.921944 5311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:37.957049 5311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:37.959458 5311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:37.959659 5311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:37.965301 5311 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:43591
I20260620 08:16:37.965324 5378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:43591 every 8 connection(s)
I20260620 08:16:37.965757 5311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:16:37.966602 5379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:37.969640 5379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Bootstrap starting.
I20260620 08:16:37.970465 5379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:37.970799 5379 log.cc:824] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:37.971640 5379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: No bootstrap required, opened a new log
I20260620 08:16:37.973747 5379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:37.973971 5379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:37.974223 5379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3ca23c265e4032a23ee074bb031ed5, State: Initialized, Role: FOLLOWER
I20260620 08:16:37.974439 5379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:37.974547 5379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:16:37.974593 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 5311
I20260620 08:16:37.974610 5379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:16:37.974668 5379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:37.974697 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:16:37.974870 25574 external_mini_cluster.cc:1428] Setting key eef79778195e66a9c66a9416b07f923e
I20260620 08:16:37.975498 5379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:37.975630 5379 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3ca23c265e4032a23ee074bb031ed5; no voters:
I20260620 08:16:37.975817 5379 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:16:37.975853 5384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:16:37.975989 5384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 LEADER]: Becoming Leader. State: Replica: 5b3ca23c265e4032a23ee074bb031ed5, State: Running, Role: LEADER
I20260620 08:16:37.976089 5384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:37.976121 5379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:37.977067 5385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b3ca23c265e4032a23ee074bb031ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } } }
I20260620 08:16:37.977164 5385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:37.977638 5386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3ca23c265e4032a23ee074bb031ed5. Latest consensus state: current_term: 1 leader_uuid: "5b3ca23c265e4032a23ee074bb031ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } } }
I20260620 08:16:37.977741 5386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:37.977841 5379 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:16:37.978142 5394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:16:37.981396 5394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:16:37 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:37.994266 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:37.975413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.24.249.190:43591)
0620 08:16:37.975655 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:37.975664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:37.975735 (+ 71us) server_negotiation.cc:407] Connection header received
0620 08:16:37.975957 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:37.975965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:37.976028 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:37.977115 (+ 1087us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:37.977344 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:37.981271 (+ 3927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:37.982570 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:37.983551 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:37.985818 (+ 2267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:37.987635 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:37.987640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:37.987641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:37.987654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:37.991666 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:37.992342 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:37.992346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:37.992347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:37.992481 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:37.992735 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:37.992736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:37.992736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:37.993113 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:37.993725 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:37.993808 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 08:16:37.993978 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260620 08:16:37.995409 5394 catalog_manager.cc:1347] Generated new cluster ID: a862ae850d1048429154c02edecd977b
I20260620 08:16:37.995476 5394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:16:38.008831 5394 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:16:38.009822 5394 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:16:38.018307 5394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Generated new TSK 0
I20260620 08:16:38.018570 5394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:16:38.729278 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:43591
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:38.822345 5424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:38.822575 5424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:38.822618 5424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:38.823796 5424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:38.823863 5424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:38.823909 5424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:16:38.825551 5424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.249.148:40653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:38.825946 5424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:38.826205 5424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:38.831246 5430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:38.831274 5429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:38.831411 5424 server_base.cc:1034] running on GCE node
W20260620 08:16:38.831295 5432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:38.831737 5424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:38.832010 5424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:38.833173 5424 hybrid_clock.cc:648] HybridClock initialized: now 1781943398833148 us; error 40 us; skew 500 ppm
Jun 20 08:16:38 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943398, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:38.836038 5424 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:16:38.836604 5424 webserver.cc:460] Webserver started at http://127.24.249.129:34929/ using document root <none> and password file <none>
I20260620 08:16:38.836896 5424 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:38.836987 5424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:38.837136 5424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:16:38 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943398, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:38.851964 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "33a4d059098a4c078ca7e2e940237a16"
format_stamp: "Formatted at 2026-06-20 08:16:38 on dist-test-slave-65w8"
server_key: "8be1b542ba73c55893350f0a57682245"
server_key_iv: "d28c6b3265bec60dc0bffa694ff6fab6"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:38.852528 5424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "33a4d059098a4c078ca7e2e940237a16"
format_stamp: "Formatted at 2026-06-20 08:16:38 on dist-test-slave-65w8"
server_key: "8be1b542ba73c55893350f0a57682245"
server_key_iv: "d28c6b3265bec60dc0bffa694ff6fab6"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:38.854048 5424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:38.874066 5439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:38.874392 5424 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 08:16:38.874537 5424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "33a4d059098a4c078ca7e2e940237a16"
format_stamp: "Formatted at 2026-06-20 08:16:38 on dist-test-slave-65w8"
server_key: "8be1b542ba73c55893350f0a57682245"
server_key_iv: "d28c6b3265bec60dc0bffa694ff6fab6"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:38.874722 5424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:38.910694 5424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:38.913020 5424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:38.913249 5424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:38.913545 5424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:38.914016 5424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:38.914124 5424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:38.914227 5424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:38.914302 5424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:38.922425 5424 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:43865
I20260620 08:16:38.922971 5424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:16:38.923564 5554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:43865 every 8 connection(s)
I20260620 08:16:38.926064 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 5424
I20260620 08:16:38.926205 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:16:38.926400 25574 external_mini_cluster.cc:1428] Setting key a1cb9f689059ef72b91f25207d42086f
Jun 20 08:16:38 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943398, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:38.936669 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:38.924036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:43591 (local address 127.24.249.129:47579)
0620 08:16:38.924385 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:38.924425 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:38.925200 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:38.925544 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:38.925548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:38.925779 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:38.926156 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:38.926164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:38.927205 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:38.927206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:38.928679 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:38.928682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:38.929201 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:38.929202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:38.929301 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:38.929928 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:38.929938 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:38.931428 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:38.934812 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:38.934821 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:38.934823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:38.935119 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:38.935357 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:38.935358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:38.935359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:38.935518 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:38.935865 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:38.935872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:38.936079 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 08:16:38.936200 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 08:16:38.936396 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":214,"threads_started":1}
I20260620 08:16:38.937506 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:38.924040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:47579 (local address 127.24.249.190:43591)
0620 08:16:38.924295 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:38.924297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:38.924506 (+ 209us) server_negotiation.cc:407] Connection header received
0620 08:16:38.925378 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:38.925379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:38.925394 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:38.925473 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:38.926284 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:38.927051 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:38.928800 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:38.929076 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:38.929112 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:38.931528 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:38.931530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:38.931530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:38.931538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:38.934715 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:38.935215 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:38.935216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:38.935217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:38.935290 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:38.935605 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:38.935606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:38.935607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:38.935766 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:38.935835 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:38.937295 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0620 08:16:38.937387 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
I20260620 08:16:38.938186 5555 heartbeater.cc:347] Connected to a master server at 127.24.249.190:43591
I20260620 08:16:38.938338 5555 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:38.938614 5555 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:16:38.939235 5343 ts_manager.cc:194] Registered new tserver with Master: 33a4d059098a4c078ca7e2e940237a16 (127.24.249.129:43865)
I20260620 08:16:38.939863 5343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:47579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:16:38.957161 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:43591
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:39.069075 5561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:39.069368 5561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:39.069406 5561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:39.071134 5561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:39.071197 5561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:39.071237 5561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:16:39.073333 5561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.249.148:40653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:39.073604 5561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:39.073938 5561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:39.080441 5571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:39.080726 5561 server_base.cc:1034] running on GCE node
W20260620 08:16:39.080952 5568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.081081 5569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:39.081357 5561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:39.081646 5561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:39.082831 5561 hybrid_clock.cc:648] HybridClock initialized: now 1781943399082792 us; error 55 us; skew 500 ppm
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:39.086093 5561 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:16:39.086642 5561 webserver.cc:460] Webserver started at http://127.24.249.130:46731/ using document root <none> and password file <none>
I20260620 08:16:39.086921 5561 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:39.086995 5561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:39.087154 5561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:39.105753 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "23931fd0b926408284b20d4048597496"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "8231f223161ffdd7ec8d86c900964273"
server_key_iv: "c48e28ee4335f99acf9b2e49ec03c2ea"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.111312 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "23931fd0b926408284b20d4048597496"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "8231f223161ffdd7ec8d86c900964273"
server_key_iv: "c48e28ee4335f99acf9b2e49ec03c2ea"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.118120 5561 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260620 08:16:39.138054 5578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.138335 5561 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 08:16:39.138439 5561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "23931fd0b926408284b20d4048597496"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "8231f223161ffdd7ec8d86c900964273"
server_key_iv: "c48e28ee4335f99acf9b2e49ec03c2ea"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.138541 5561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:39.155359 5561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:39.158296 5561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:39.158447 5561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:39.158660 5561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:39.159081 5561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:39.159127 5561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.159158 5561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:39.159222 5561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.165668 5561 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:35475
I20260620 08:16:39.166908 5561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:16:39.169103 5691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:35475 every 8 connection(s)
I20260620 08:16:39.170125 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 5561
I20260620 08:16:39.170229 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:16:39.170418 25574 external_mini_cluster.cc:1428] Setting key a81bd8093c35d7fdc6a7ace32abc6859
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:39.179034 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:39.167777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:43591 (local address 127.24.249.130:43315)
0620 08:16:39.167995 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:39.168049 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:39.168693 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:39.169203 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:39.169208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:39.169479 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:39.169777 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:39.169783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.170854 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.170855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:39.172094 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:39.172097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.172677 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.172678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:39.172816 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:39.173453 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:39.173464 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:39.175111 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:39.177227 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:39.177235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:39.177237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:39.177562 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:39.177783 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:39.177784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:39.177784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:39.177959 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:39.178254 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:39.178260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:39.178421 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 08:16:39.178556 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 08:16:39.178741 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260620 08:16:39.179145 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:39.168067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:43315 (local address 127.24.249.190:43591)
0620 08:16:39.168116 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:39.168118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:39.168130 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:16:39.168826 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:39.168828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:39.168842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:39.168876 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:39.169962 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.170741 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.172195 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.172510 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.172593 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:39.175219 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:39.175221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:39.175221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:39.175229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:39.177126 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:39.177654 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:39.177656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:39.177656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:39.177717 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:39.178033 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:39.178034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:39.178034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:39.178173 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:39.178231 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:39.178934 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 08:16:39.179035 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 08:16:39.179539 5692 heartbeater.cc:347] Connected to a master server at 127.24.249.190:43591
I20260620 08:16:39.179637 5692 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:39.179819 5692 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:39.180169 5343 ts_manager.cc:194] Registered new tserver with Master: 23931fd0b926408284b20d4048597496 (127.24.249.130:35475)
I20260620 08:16:39.180629 5343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:43315
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:16:39.201593 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:43591
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:39.292285 5698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:39.292510 5698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:39.292547 5698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:39.293715 5698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:39.293782 5698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:39.293889 5698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:16:39.295382 5698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.249.148:40653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:39.295650 5698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:39.295871 5698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:39.301258 5706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.301324 5704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.301465 5703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:39.301646 5698 server_base.cc:1034] running on GCE node
I20260620 08:16:39.301970 5698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:39.302230 5698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:39.303376 5698 hybrid_clock.cc:648] HybridClock initialized: now 1781943399303355 us; error 38 us; skew 500 ppm
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:39.306162 5698 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:16:39.306723 5698 webserver.cc:460] Webserver started at http://127.24.249.131:39495/ using document root <none> and password file <none>
I20260620 08:16:39.306958 5698 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:39.307027 5698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:39.307171 5698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:39.325790 5698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "190d0bcae41a7244b89194a3cfa677f8"
server_key_iv: "15011ac53f83c549326c7e4c0d9dc8f5"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.326313 5698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "190d0bcae41a7244b89194a3cfa677f8"
server_key_iv: "15011ac53f83c549326c7e4c0d9dc8f5"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.330070 5698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 08:16:39.354019 5713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.354246 5698 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260620 08:16:39.354352 5698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "190d0bcae41a7244b89194a3cfa677f8"
server_key_iv: "15011ac53f83c549326c7e4c0d9dc8f5"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.354476 5698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:39.381150 5698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:39.383630 5698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:39.383772 5698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:39.384024 5698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:39.384382 5698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:39.384428 5698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.384459 5698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:39.384482 5698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.391736 5698 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:38541
I20260620 08:16:39.392204 5698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:16:39.392764 5826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:38541 every 8 connection(s)
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:39.402436 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 5698
I20260620 08:16:39.402580 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:16:39.402838 25574 external_mini_cluster.cc:1428] Setting key 332721e0ce30586e92bbbe89e58c5dd2
I20260620 08:16:39.405140 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:39.393326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:60585 (local address 127.24.249.190:43591)
0620 08:16:39.393377 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:39.393379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:39.393390 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:16:39.393978 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:39.393979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:39.393993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:39.394091 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:39.394791 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.395634 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.397136 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.397464 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.397549 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:39.400299 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:39.400301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:39.400301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:39.400309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:39.402142 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:39.403758 (+ 1616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:39.403760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:39.403761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:39.403846 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:39.404281 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:39.404283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:39.404283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:39.404433 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:39.404514 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:39.404948 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 08:16:39.405040 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:16:39.405467 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:39.392996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:43591 (local address 127.24.249.131:60585)
0620 08:16:39.393161 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:39.393202 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:39.393825 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:39.394162 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:39.394166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:39.394407 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:39.394679 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:39.394685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.395751 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.395752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:39.397033 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:39.397036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.397574 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.397575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:39.397723 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:39.398426 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:39.398436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:39.400189 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:39.403066 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:39.403076 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:39.403080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:39.403619 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:39.403981 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:39.403982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:39.403982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:39.404190 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:39.404526 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:39.404534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:39.404756 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 08:16:39.404912 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 08:16:39.405148 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260620 08:16:39.406116 5827 heartbeater.cc:347] Connected to a master server at 127.24.249.190:43591
I20260620 08:16:39.406242 5827 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:39.406479 5827 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:39.406915 5343 ts_manager.cc:194] Registered new tserver with Master: 3ecffeefbb5d49aa8e5cd0b965b6f149 (127.24.249.131:38541)
I20260620 08:16:39.407439 5343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:60585
I20260620 08:16:39.414714 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:16:39.415458 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 5424
I20260620 08:16:39.424674 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 5561
I20260620 08:16:39.430909 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 5698
I20260620 08:16:39.437373 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 5311
I20260620 08:16:39.444110 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43591
--webserver_interface=127.24.249.190
--webserver_port=33561
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:43591
--ranger_config_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:39.531760 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:39.531984 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:39.532006 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:39.533192 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:39.533237 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:39.533252 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:39.533264 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:39.533277 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:39.534904 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:43591
--ranger_config_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=33561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:39.535140 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:39.535377 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:39.540575 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.540575 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.540647 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:39.540694 5830 server_base.cc:1034] running on GCE node
I20260620 08:16:39.541117 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:39.541450 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:39.542624 5830 hybrid_clock.cc:648] HybridClock initialized: now 1781943399542584 us; error 58 us; skew 500 ppm
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:39.545246 5830 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:16:39.545711 5830 webserver.cc:460] Webserver started at http://127.24.249.190:33561/ using document root <none> and password file <none>
I20260620 08:16:39.546015 5830 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:39.546093 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:39.596083 5830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:39.596879 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.597087 5830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:39.597180 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "5b3ca23c265e4032a23ee074bb031ed5"
format_stamp: "Formatted at 2026-06-20 08:16:37 on dist-test-slave-65w8"
server_key: "c4ddbd5233744c83ec40be3c9a55b814"
server_key_iv: "3a4e1f262b86deb88ba677c774082e14"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.597560 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:39.614773 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:39.616798 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:39.616979 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:39.621852 5830 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:43591
I20260620 08:16:39.621917 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:43591 every 8 connection(s)
I20260620 08:16:39.622291 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:16:39.623785 5899 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:16:39.624497 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Bootstrap starting.
I20260620 08:16:39.630371 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 5830
I20260620 08:16:39.630673 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:43865
--local_ip_for_outbound_sockets=127.24.249.129
--tserver_master_addrs=127.24.249.190:43591
--webserver_port=34929
--webserver_interface=127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:16:39.633682 5899 log.cc:824] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:39.635398 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:16:39.635710 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Bootstrap complete.
I20260620 08:16:39.638348 5899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:39.638612 5899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3ca23c265e4032a23ee074bb031ed5, State: Initialized, Role: FOLLOWER
I20260620 08:16:39.638815 5899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:39.638939 5899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:16:39.639015 5899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:16:39.639068 5899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:16:39.640101 5899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:39.640226 5899 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3ca23c265e4032a23ee074bb031ed5; no voters:
I20260620 08:16:39.640406 5899 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:16:39.640479 5903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:16:39.640626 5903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [term 2 LEADER]: Becoming Leader. State: Replica: 5b3ca23c265e4032a23ee074bb031ed5, State: Running, Role: LEADER
I20260620 08:16:39.640691 5899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:39.640718 5903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } }
I20260620 08:16:39.640949 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3ca23c265e4032a23ee074bb031ed5. Latest consensus state: current_term: 2 leader_uuid: "5b3ca23c265e4032a23ee074bb031ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } } }
I20260620 08:16:39.641103 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:39.640950 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b3ca23c265e4032a23ee074bb031ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3ca23c265e4032a23ee074bb031ed5" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43591 } } }
I20260620 08:16:39.641455 5909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:16:39.642041 5909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:16:39.642373 5899 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 08:16:39.641450 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:39.642817 5909 catalog_manager.cc:1259] Loaded cluster ID: a862ae850d1048429154c02edecd977b
I20260620 08:16:39.642920 5909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:16:39.646883 5909 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:16:39.647311 5909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Loaded TSK: 0
I20260620 08:16:39.647598 5909 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:16:39.745962 5901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:39.746237 5901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:39.746279 5901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:39.748059 5901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:39.748193 5901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:39.748286 5901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:16:39.750563 5901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:43865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=34929
--tserver_master_addrs=127.24.249.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:39.751062 5901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:39.751399 5901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:39.758145 5937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.758404 5939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:39.758682 5901 server_base.cc:1034] running on GCE node
W20260620 08:16:39.758812 5936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:39.759066 5901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:39.759348 5901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:39.760517 5901 hybrid_clock.cc:648] HybridClock initialized: now 1781943399760495 us; error 36 us; skew 500 ppm
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:39.764184 5901 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:16:39.764770 5901 webserver.cc:460] Webserver started at http://127.24.249.129:34929/ using document root <none> and password file <none>
I20260620 08:16:39.765058 5901 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:39.765148 5901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:39.791893 5901 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 08:16:39.793040 5947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.793258 5901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:16:39.793339 5901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "33a4d059098a4c078ca7e2e940237a16"
format_stamp: "Formatted at 2026-06-20 08:16:38 on dist-test-slave-65w8"
server_key: "8be1b542ba73c55893350f0a57682245"
server_key_iv: "d28c6b3265bec60dc0bffa694ff6fab6"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.793797 5901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:39.819871 5901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:39.822549 5901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:39.822777 5901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:39.823117 5901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:39.823556 5901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:39.823647 5901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.823745 5901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:39.823817 5901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.832278 5901 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:43865
I20260620 08:16:39.832815 5901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:16:39.833204 6060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:43865 every 8 connection(s)
I20260620 08:16:39.833968 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 5901
I20260620 08:16:39.834290 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:35475
--local_ip_for_outbound_sockets=127.24.249.130
--tserver_master_addrs=127.24.249.190:43591
--webserver_port=46731
--webserver_interface=127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:39.853111 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:39.834043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:43591 (local address 127.24.249.129:52857)
0620 08:16:39.834588 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:39.834629 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:39.835230 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:39.835952 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:39.835957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:39.836204 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:39.836481 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:39.836487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.838053 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.838054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:39.839226 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:39.839229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.840093 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.840095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:39.840237 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:39.840888 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:39.840898 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:39.842690 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:39.851092 (+ 8402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:39.851104 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:39.851107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:39.851445 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:39.851771 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:39.851772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:39.851773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:39.851920 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:39.852435 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:39.852442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:39.852596 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 08:16:39.852689 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 08:16:39.852855 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":93,"threads_started":1}
I20260620 08:16:39.854207 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:39.834066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:52857 (local address 127.24.249.190:43591)
0620 08:16:39.835449 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:39.835459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:39.835502 (+ 43us) server_negotiation.cc:407] Connection header received
0620 08:16:39.835645 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:39.835652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:39.835707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:39.836616 (+ 909us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:39.836696 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.837926 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.839341 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:39.839948 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:39.842844 (+ 2896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:39.842940 (+ 96us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:39.842944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:39.842944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:39.842953 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:39.850947 (+ 7994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:39.851542 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:39.851545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:39.851546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:39.851684 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:39.851987 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:39.851987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:39.851988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:39.852303 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:39.853763 (+ 1460us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:39.853840 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 08:16:39.853998 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1306,"thread_start_us":71,"threads_started":1}
I20260620 08:16:39.855371 6061 heartbeater.cc:347] Connected to a master server at 127.24.249.190:43591
I20260620 08:16:39.855465 6061 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:39.855657 6061 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
W20260620 08:16:39.856552 6061 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:43591 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:16:39.856621 6061 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:39.856678 6061 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:39.857108 6061 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:39.857192 6061 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
W20260620 08:16:39.857564 6061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:16:39.944649 6065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:39.944875 6065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:39.944916 6065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:39.946280 6065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:39.946346 6065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:39.946393 6065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:16:39.948025 6065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:35475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=46731
--tserver_master_addrs=127.24.249.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:39.948290 6065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:39.948609 6065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:39.953575 6073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.953722 6070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:39.953789 6071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:39.954144 6065 server_base.cc:1034] running on GCE node
I20260620 08:16:39.954341 6065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:39.954604 6065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:39.955787 6065 hybrid_clock.cc:648] HybridClock initialized: now 1781943399955766 us; error 34 us; skew 500 ppm
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:39.958621 6065 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:16:39.959131 6065 webserver.cc:460] Webserver started at http://127.24.249.130:46731/ using document root <none> and password file <none>
I20260620 08:16:39.959372 6065 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:39.959450 6065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:16:39 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:39.975224 6065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:39.976310 6081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.976565 6065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:39.976668 6065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "23931fd0b926408284b20d4048597496"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "8231f223161ffdd7ec8d86c900964273"
server_key_iv: "c48e28ee4335f99acf9b2e49ec03c2ea"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:39.977149 6065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:39.991781 6065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:39.994131 6065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:39.994283 6065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:39.994550 6065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:39.994957 6065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:39.995029 6065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:39.995064 6065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:39.995298 6065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:40.002820 6065 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:35475
I20260620 08:16:40.003057 6194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:35475 every 8 connection(s)
I20260620 08:16:40.003321 6065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:16:40.005160 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6065
I20260620 08:16:40.005477 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.249.148:40653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:38541
--local_ip_for_outbound_sockets=127.24.249.131
--tserver_master_addrs=127.24.249.190:43591
--webserver_port=39495
--webserver_interface=127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:16:40 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943399, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:40.016194 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:40.004510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:52291 (local address 127.24.249.190:43591)
0620 08:16:40.004574 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:40.004576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:40.005426 (+ 850us) server_negotiation.cc:407] Connection header received
0620 08:16:40.006173 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:40.006175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:40.006188 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:40.006244 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:40.007052 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.007653 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.008816 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.009055 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.009148 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:40.011928 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:40.011930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:40.011931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:40.011941 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:40.014103 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:40.014756 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:40.014758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:40.014758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:40.014826 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:40.015169 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:40.015170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:40.015171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:40.015354 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:40.015410 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:40.015925 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 08:16:40.016032 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:16:40.017201 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:40.004195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:43591 (local address 127.24.249.130:52291)
0620 08:16:40.005282 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:40.005329 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:40.006058 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:40.006327 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:40.006331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:40.006518 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:40.006789 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:40.006794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.007793 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.007795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:40.008696 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:40.008698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.009187 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.009188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:40.009319 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:40.009980 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:40.009992 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:40.011812 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:40.014222 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:40.014231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:40.014235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:40.014645 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:40.014916 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:40.014917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:40.014918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:40.015075 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:40.015473 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:40.015481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:40.015706 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 08:16:40.016661 (+ 955us) client_negotiation.cc:240] Negotiation successful
0620 08:16:40.016878 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"thread_start_us":90,"threads_started":1}
I20260620 08:16:40.018132 6195 heartbeater.cc:347] Connected to a master server at 127.24.249.190:43591
I20260620 08:16:40.018308 6195 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:40.018602 6195 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
W20260620 08:16:40.019373 6195 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:43591 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:16:40.019526 6195 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:40.019656 6195 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:40.020058 6195 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:40.020198 6195 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
W20260620 08:16:40.020589 6195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:16:40.117599 6198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:40.117833 6198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:40.117872 6198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:40.119269 6198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:40.119338 6198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:40.119385 6198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:16:40.120972 6198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.249.148:40653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:38541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=39495
--tserver_master_addrs=127.24.249.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:40.121234 6198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:40.121464 6198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:40.126753 6203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:40.127166 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:40.128175 6198 server_base.cc:1034] running on GCE node
W20260620 08:16:40.128671 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:40.128952 6198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:40.129261 6198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:40.130445 6198 hybrid_clock.cc:648] HybridClock initialized: now 1781943400130412 us; error 46 us; skew 500 ppm
Jun 20 08:16:40 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:40.134497 6198 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:16:40.135000 6198 webserver.cc:460] Webserver started at http://127.24.249.131:39495/ using document root <none> and password file <none>
I20260620 08:16:40.135249 6198 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:40.135336 6198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 08:16:40 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:40.153153 6198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:16:40.155329 6215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:40.155561 6198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:16:40.155658 6198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149"
format_stamp: "Formatted at 2026-06-20 08:16:39 on dist-test-slave-65w8"
server_key: "190d0bcae41a7244b89194a3cfa677f8"
server_key_iv: "15011ac53f83c549326c7e4c0d9dc8f5"
server_key_version: "kuduclusterkey@0"
I20260620 08:16:40.156073 6198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:40.169589 6198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:40.171797 6198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:40.171975 6198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:40.172245 6198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:40.172631 6198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:40.172686 6198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:40.172726 6198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:40.172751 6198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:40.180112 6198 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:38541
I20260620 08:16:40.180194 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:38541 every 8 connection(s)
I20260620 08:16:40.180533 6198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:16:40 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:40.190543 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6198
I20260620 08:16:40.191321 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:40.181316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:41617 (local address 127.24.249.190:43591)
0620 08:16:40.181371 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:40.181374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:40.181610 (+ 236us) server_negotiation.cc:407] Connection header received
0620 08:16:40.182256 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:40.182257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:40.182269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:40.182323 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:40.182988 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.183590 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.185012 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.185212 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.185272 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:40.187741 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:40.187742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:40.187743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:40.187750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:40.189355 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:40.189834 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:40.189835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:40.189836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:40.189919 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:40.190198 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:40.190199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:40.190200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:40.190320 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:40.190858 (+ 538us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:40.191139 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 08:16:40.191234 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:16:40.191752 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:40.181292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:43591 (local address 127.24.249.131:41617)
0620 08:16:40.181465 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:40.181507 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:40.182150 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:40.182390 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:40.182394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:40.182628 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:40.182883 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:40.182888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.183686 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.183687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:40.184922 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:40.184924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.185328 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.185330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:40.185473 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:40.186105 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:40.186116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:40.187626 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:40.189464 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:40.189472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:40.189474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:40.189735 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:40.189994 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:40.189995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:40.189995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:40.190120 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:40.190850 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:40.190856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:40.190998 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 08:16:40.191106 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 08:16:40.191253 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260620 08:16:40.192279 6329 heartbeater.cc:347] Connected to a master server at 127.24.249.190:43591
I20260620 08:16:40.192396 6329 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:40.192608 6329 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
W20260620 08:16:40.193516 6329 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:43591 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 08:16:40.193614 6329 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:40.193689 6329 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:40.194101 6329 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:40.194187 6329 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
W20260620 08:16:40.194523 6329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:16:40.202334 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:40.192509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.24.249.190:43591)
0620 08:16:40.192560 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:40.192562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:40.192571 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:16:40.192592 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:40.192593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:40.192603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:40.192664 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:40.196386 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.196885 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.197739 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:40.198022 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:40.198110 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:40.199481 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:40.199482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:40.199483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:40.199489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:40.201064 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:40.201458 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:40.201459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:40.201460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:40.201521 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:40.201761 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:40.201762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:40.201762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:40.201929 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:40.202119 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:40.202157 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 08:16:40.202232 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 08:16:40 dist-test-slave-65w8 krb5kdc[4867](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781943400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:40.857758 6061 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:40.857949 6061 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:40.858371 5862 ts_manager.cc:194] Registered new tserver with Master: 33a4d059098a4c078ca7e2e940237a16 (127.24.249.129:43865)
I20260620 08:16:40.859391 5862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:52857
I20260620 08:16:41.020840 6195 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:41.021018 6195 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:41.021498 5862 ts_manager.cc:194] Registered new tserver with Master: 23931fd0b926408284b20d4048597496 (127.24.249.130:35475)
I20260620 08:16:41.022156 5862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:52291
Jun 20 08:16:41 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:41.194738 6329 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:41.194905 6329 heartbeater.cc:511] Master 127.24.249.190:43591 requested a full tablet report, sending...
I20260620 08:16:41.195335 5862 ts_manager.cc:194] Registered new tserver with Master: 3ecffeefbb5d49aa8e5cd0b965b6f149 (127.24.249.131:38541)
I20260620 08:16:41.195822 5862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:41617
I20260620 08:16:41.197307 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:16:41.207103 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.198295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.24.249.190:43591)
0620 08:16:41.198556 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:41.198558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:41.198719 (+ 161us) server_negotiation.cc:407] Connection header received
0620 08:16:41.198742 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:41.198743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:41.198755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:41.198807 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:41.199578 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.200388 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.201658 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.202025 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.202089 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.203778 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:41.203780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:41.203781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:41.203790 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:41.205618 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:41.206051 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:41.206052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:41.206052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:41.206110 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:41.206286 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:41.206287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:41.206287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:41.206408 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:41.206492 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:41.206910 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 08:16:41.207002 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":79,"threads_started":1}
I20260620 08:16:41.209429 5862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:16:41.211642 5862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:16:41.233841 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.224757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:38541 (local address 127.0.0.1:37322)
0620 08:16:41.225809 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:41.225854 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:41.225941 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:41.226298 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:41.226301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:41.226303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:41.226377 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.226378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.231046 (+ 4668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.231047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.232127 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.232130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.233524 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.233525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.233608 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.233622 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:16:41.233692 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:16:41.233746 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":81,"threads_started":1}
I20260620 08:16:41.236102 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.225177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.24.249.129:43865)
0620 08:16:41.225746 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:41.225752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:41.226562 (+ 810us) server_negotiation.cc:407] Connection header received
0620 08:16:41.226598 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:41.226600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:41.226615 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:41.226677 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:41.226868 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.228250 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.234530 (+ 6280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.235299 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.235378 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.235436 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:41.235980 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 08:16:41.236026 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":51,"threads_started":1}
I20260620 08:16:41.236455 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.224983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:43865 (local address 127.0.0.1:45158)
0620 08:16:41.226466 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:41.226475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:41.226511 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:41.226755 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:41.226756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:41.226758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:41.226800 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.226801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.228345 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.228347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.234448 (+ 6101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.234449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.235473 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.235473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.235534 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.235538 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:16:41.235584 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:16:41.235626 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"thread_start_us":56,"threads_started":1}
I20260620 08:16:41.236877 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.224864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.24.249.131:38541)
0620 08:16:41.226088 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:41.226092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:41.226101 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:16:41.226133 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:41.226136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:41.226151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:41.226214 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:41.229831 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.230926 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.232588 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.233355 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.233426 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.233488 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:41.236753 (+ 3265us) server_negotiation.cc:299] Negotiation successful
0620 08:16:41.236806 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1193,"thread_start_us":59,"threads_started":1}
I20260620 08:16:41.237367 5995 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae241e3c9ba4b9a92c299d2b350d391 (DEFAULT_TABLE table=test-table [id=83ce321a65f342b4b4fe0b2b0abe6d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:41.237651 6251 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae241e3c9ba4b9a92c299d2b350d391 (DEFAULT_TABLE table=test-table [id=83ce321a65f342b4b4fe0b2b0abe6d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:41.237789 5995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae241e3c9ba4b9a92c299d2b350d391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:41.238018 6251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae241e3c9ba4b9a92c299d2b350d391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:41.245682 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.225219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:35475 (local address 127.0.0.1:54222)
0620 08:16:41.234326 (+ 9107us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:41.234337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:41.234369 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:41.241902 (+ 7533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:41.241903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:41.241905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:41.241947 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.241948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.243176 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.243177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.244304 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.244307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.245433 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.245434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.245499 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.245503 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:16:41.245551 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 08:16:41.245598 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9094,"thread_start_us":90,"threads_started":1}
I20260620 08:16:41.246237 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.225317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.24.249.130:35475)
0620 08:16:41.236481 (+ 11164us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:41.236488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:41.236500 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:16:41.236541 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:41.236544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:41.236561 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:41.236659 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:41.242024 (+ 5365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.243060 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.244409 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.245270 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.245343 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.245399 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:41.246102 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 08:16:41.246159 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11120,"thread_start_us":53,"threads_started":1}
I20260620 08:16:41.246727 6362 tablet_bootstrap.cc:492] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Bootstrap starting.
I20260620 08:16:41.247078 6118 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae241e3c9ba4b9a92c299d2b350d391 (DEFAULT_TABLE table=test-table [id=83ce321a65f342b4b4fe0b2b0abe6d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:41.247467 6118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae241e3c9ba4b9a92c299d2b350d391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:41.247740 6362 tablet_bootstrap.cc:654] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:41.248049 6362 log.cc:824] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:41.248484 6363 tablet_bootstrap.cc:492] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16: Bootstrap starting.
I20260620 08:16:41.249207 6363 tablet_bootstrap.cc:654] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:41.249557 6363 log.cc:824] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:41.250452 6362 tablet_bootstrap.cc:492] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: No bootstrap required, opened a new log
I20260620 08:16:41.250536 6362 ts_tablet_manager.cc:1397] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260620 08:16:41.254298 6362 raft_consensus.cc:348] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.254698 6362 raft_consensus.cc:374] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:41.254798 6362 raft_consensus.cc:729] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ecffeefbb5d49aa8e5cd0b965b6f149, State: Initialized, Role: FOLLOWER
I20260620 08:16:41.255064 6362 consensus_queue.cc:260] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.255581 6362 ts_tablet_manager.cc:1428] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 08:16:41.256536 6363 tablet_bootstrap.cc:492] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16: No bootstrap required, opened a new log
I20260620 08:16:41.256677 6363 ts_tablet_manager.cc:1397] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 08:16:41.259429 6329 heartbeater.cc:503] Master 127.24.249.190:43591 was elected leader, sending a full tablet report...
I20260620 08:16:41.259922 6363 raft_consensus.cc:348] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.260180 6363 raft_consensus.cc:374] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:41.260263 6363 raft_consensus.cc:729] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a4d059098a4c078ca7e2e940237a16, State: Initialized, Role: FOLLOWER
I20260620 08:16:41.260453 6363 consensus_queue.cc:260] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.260648 6366 tablet_bootstrap.cc:492] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Bootstrap starting.
I20260620 08:16:41.260907 6363 ts_tablet_manager.cc:1428] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:16:41.261057 6061 heartbeater.cc:503] Master 127.24.249.190:43591 was elected leader, sending a full tablet report...
I20260620 08:16:41.261412 6366 tablet_bootstrap.cc:654] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:41.261727 6366 log.cc:824] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:41.263041 6366 tablet_bootstrap.cc:492] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: No bootstrap required, opened a new log
I20260620 08:16:41.263108 6366 ts_tablet_manager.cc:1397] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:16:41.265583 6366 raft_consensus.cc:348] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.265733 6366 raft_consensus.cc:374] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:41.265762 6366 raft_consensus.cc:729] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23931fd0b926408284b20d4048597496, State: Initialized, Role: FOLLOWER
I20260620 08:16:41.266144 6366 consensus_queue.cc:260] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.266469 6366 ts_tablet_manager.cc:1428] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:16:41.266644 6195 heartbeater.cc:503] Master 127.24.249.190:43591 was elected leader, sending a full tablet report...
W20260620 08:16:41.335495 6062 tablet.cc:2369] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:16:41.366497 6370 raft_consensus.cc:482] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:41.366699 6370 raft_consensus.cc:504] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.368091 6370 leader_election.cc:288] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ecffeefbb5d49aa8e5cd0b965b6f149 (127.24.249.131:38541), 33a4d059098a4c078ca7e2e940237a16 (127.24.249.129:43865)
I20260620 08:16:41.372012 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.368031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:40203 (local address 127.24.249.129:43865)
0620 08:16:41.368072 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:41.368074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:41.368326 (+ 252us) server_negotiation.cc:407] Connection header received
0620 08:16:41.368429 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:41.368430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:41.368439 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:41.368469 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:41.368669 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.369381 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.371030 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.371577 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.371632 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.371642 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:41.371883 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 08:16:41.371927 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:16:41.372453 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.367850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:43865 (local address 127.24.249.130:40203)
0620 08:16:41.368262 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:41.368271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:41.368353 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:41.368531 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:41.368532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:41.368533 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:41.368602 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.368604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.369685 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.369686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.370902 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.370905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.371724 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.371725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.371795 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.371799 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:16:41.372316 (+ 517us) client_negotiation.cc:240] Negotiation successful
0620 08:16:41.372359 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":77,"threads_started":1}
I20260620 08:16:41.372938 6012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae241e3c9ba4b9a92c299d2b350d391" candidate_uuid: "23931fd0b926408284b20d4048597496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a4d059098a4c078ca7e2e940237a16" is_pre_election: true
I20260620 08:16:41.373121 6012 raft_consensus.cc:2454] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23931fd0b926408284b20d4048597496 in term 0.
I20260620 08:16:41.373423 6082 leader_election.cc:302] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23931fd0b926408284b20d4048597496, 33a4d059098a4c078ca7e2e940237a16; no voters:
I20260620 08:16:41.373663 6370 raft_consensus.cc:2781] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:16:41.373726 6370 raft_consensus.cc:482] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:16:41.373754 6370 raft_consensus.cc:3037] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:41.374032 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.367659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:46919 (local address 127.24.249.131:38541)
0620 08:16:41.367706 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:41.367708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:41.367931 (+ 223us) server_negotiation.cc:407] Connection header received
0620 08:16:41.368183 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:41.368184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:41.368195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:41.368223 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:41.368440 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.369139 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.373112 (+ 3973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.373634 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.373677 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.373684 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:41.373922 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 08:16:41.373970 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:16:41.374409 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.367439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:38541 (local address 127.24.249.130:46919)
0620 08:16:41.367840 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:41.367854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:41.367962 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:41.368294 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:41.368294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:41.368297 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:41.368372 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.368374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.370277 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.370278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.373005 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:41.373008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.373738 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.373739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:41.373805 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.373808 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:16:41.374289 (+ 481us) client_negotiation.cc:240] Negotiation successful
0620 08:16:41.374332 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":96,"threads_started":1}
I20260620 08:16:41.374826 6279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae241e3c9ba4b9a92c299d2b350d391" candidate_uuid: "23931fd0b926408284b20d4048597496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" is_pre_election: true
I20260620 08:16:41.374927 6370 raft_consensus.cc:504] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.375000 6279 raft_consensus.cc:2454] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23931fd0b926408284b20d4048597496 in term 0.
I20260620 08:16:41.375075 6370 leader_election.cc:288] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [CANDIDATE]: Term 1 election: Requested vote from peers 3ecffeefbb5d49aa8e5cd0b965b6f149 (127.24.249.131:38541), 33a4d059098a4c078ca7e2e940237a16 (127.24.249.129:43865)
I20260620 08:16:41.375236 6012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae241e3c9ba4b9a92c299d2b350d391" candidate_uuid: "23931fd0b926408284b20d4048597496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a4d059098a4c078ca7e2e940237a16"
I20260620 08:16:41.375301 6012 raft_consensus.cc:3037] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:41.375339 6279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae241e3c9ba4b9a92c299d2b350d391" candidate_uuid: "23931fd0b926408284b20d4048597496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149"
I20260620 08:16:41.375398 6279 raft_consensus.cc:3037] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:41.376164 6012 raft_consensus.cc:2454] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23931fd0b926408284b20d4048597496 in term 1.
I20260620 08:16:41.376209 6279 raft_consensus.cc:2454] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23931fd0b926408284b20d4048597496 in term 1.
I20260620 08:16:41.376412 6082 leader_election.cc:302] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23931fd0b926408284b20d4048597496, 3ecffeefbb5d49aa8e5cd0b965b6f149; no voters:
I20260620 08:16:41.377940 6370 raft_consensus.cc:2781] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:16:41.378127 6370 raft_consensus.cc:686] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 1 LEADER]: Becoming Leader. State: Replica: 23931fd0b926408284b20d4048597496, State: Running, Role: LEADER
I20260620 08:16:41.378249 6370 consensus_queue.cc:237] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } }
I20260620 08:16:41.379150 5862 catalog_manager.cc:5495] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23931fd0b926408284b20d4048597496 (127.24.249.130). New cstate: current_term: 1 leader_uuid: "23931fd0b926408284b20d4048597496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23931fd0b926408284b20d4048597496" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 35475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 } health_report { overall_health: UNKNOWN } } }
I20260620 08:16:41.414424 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.410858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.24.249.130:35475)
0620 08:16:41.410921 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:41.410924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:41.410935 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:16:41.410982 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:41.410983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:41.411007 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:41.411094 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:16:41.411301 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.412051 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.413197 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:41.413504 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:41.413568 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:41.413786 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:16:41.413986 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:16:41.414070 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:41.414314 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 08:16:41.414339 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:16:41.418223 6012 raft_consensus.cc:1261] T 2ae241e3c9ba4b9a92c299d2b350d391 P 33a4d059098a4c078ca7e2e940237a16 [term 1 FOLLOWER]: Refusing update from remote peer 23931fd0b926408284b20d4048597496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:41.418690 6370 consensus_queue.cc:1035] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a4d059098a4c078ca7e2e940237a16" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 20 08:16:41 dist-test-slave-65w8 krb5kdc[4867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943400, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for HTTP/127.24.249.148@KRBTEST.COM
I20260620 08:16:41.421622 6279 raft_consensus.cc:1261] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 1 FOLLOWER]: Refusing update from remote peer 23931fd0b926408284b20d4048597496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:41.422052 6370 consensus_queue.cc:1035] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ecffeefbb5d49aa8e5cd0b965b6f149" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 08:16:41.431880 6330 tablet.cc:2369] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:16:41.432996 6377 mvcc.cc:205] Tried to move back new op lower bound from 7298840172205375488 to 7298840172046077952. Current Snapshot: MvccSnapshot[applied={T|T < 7298840172205375488}]
I20260620 08:16:41.435590 6378 mvcc.cc:205] Tried to move back new op lower bound from 7298840172205375488 to 7298840172046077952. Current Snapshot: MvccSnapshot[applied={T|T < 7298840172205375488}]
I20260620 08:16:41.435815 6380 mvcc.cc:205] Tried to move back new op lower bound from 7298840172205375488 to 7298840172046077952. Current Snapshot: MvccSnapshot[applied={T|T < 7298840172205375488}]
I20260620 08:16:41.441772 5862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:16:41.441920 5862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:16:41.442847 5862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b3ca23c265e4032a23ee074bb031ed5: Sending DeleteTablet for 3 replicas of tablet 2ae241e3c9ba4b9a92c299d2b350d391
I20260620 08:16:41.443528 6251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae241e3c9ba4b9a92c299d2b350d391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:41 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:37322
I20260620 08:16:41.443799 5995 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae241e3c9ba4b9a92c299d2b350d391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:41 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:45158
I20260620 08:16:41.443847 6387 tablet_replica.cc:330] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: stopping tablet replica
I20260620 08:16:41.443966 6387 raft_consensus.cc:2229] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:16:41.444077 6387 raft_consensus.cc:2258] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:41.444258 6118 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae241e3c9ba4b9a92c299d2b350d391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:41 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:54222
I20260620 08:16:41.444490 6389 tablet_replica.cc:330] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: stopping tablet replica
I20260620 08:16:41.444607 6389 raft_consensus.cc:2229] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:16:41.444742 6389 raft_consensus.cc:2258] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:41.445108 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 5901
I20260620 08:16:41.445205 6389 ts_tablet_manager.cc:1905] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:16:41.447238 6389 ts_tablet_manager.cc:1918] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:16:41.447343 6389 log.cc:1196] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/2ae241e3c9ba4b9a92c299d2b350d391
I20260620 08:16:41.447659 6389 ts_tablet_manager.cc:1939] T 2ae241e3c9ba4b9a92c299d2b350d391 P 23931fd0b926408284b20d4048597496: Deleting consensus metadata
I20260620 08:16:41.449527 5850 catalog_manager.cc:4841] TS 23931fd0b926408284b20d4048597496 (127.24.249.130:35475): tablet 2ae241e3c9ba4b9a92c299d2b350d391 (table test-table [id=83ce321a65f342b4b4fe0b2b0abe6d30]) successfully deleted
I20260620 08:16:41.450145 6387 ts_tablet_manager.cc:1905] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:16:41.451844 6387 ts_tablet_manager.cc:1918] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:16:41.453122 6387 log.cc:1196] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/2ae241e3c9ba4b9a92c299d2b350d391
I20260620 08:16:41.453481 6387 ts_tablet_manager.cc:1939] T 2ae241e3c9ba4b9a92c299d2b350d391 P 3ecffeefbb5d49aa8e5cd0b965b6f149: Deleting consensus metadata
I20260620 08:16:41.453991 5847 catalog_manager.cc:4841] TS 3ecffeefbb5d49aa8e5cd0b965b6f149 (127.24.249.131:38541): tablet 2ae241e3c9ba4b9a92c299d2b350d391 (table test-table [id=83ce321a65f342b4b4fe0b2b0abe6d30]) successfully deleted
W20260620 08:16:41.459254 5847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:43865 (error 108)
I20260620 08:16:41.459498 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6065
I20260620 08:16:41.463066 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:41.462904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:43865 (local address 127.0.0.1:45162)
0620 08:16:41.462963 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:41.463019 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:16:41.463336 5847 catalog_manager.cc:4584] TS 33a4d059098a4c078ca7e2e940237a16 (127.24.249.129:43865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ae241e3c9ba4b9a92c299d2b350d391: Network error: Client connection negotiation failed: client connection to 127.24.249.129:43865: connect: Connection refused (error 111)
I20260620 08:16:41.469657 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6198
I20260620 08:16:41.476622 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 5830
I20260620 08:16:41.485348 25574 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 08:16:41.830721 25574 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 08:16:41.830847 25574 mini_ranger.cc:66] Stopping Ranger...
I20260620 08:16:42.175757 25574 mini_ranger.cc:68] Stopped Ranger
2026-06-20 08:16:42.175 UTC [4897] LOG: received smart shutdown request
2026-06-20 08:16:42.179 UTC [4897] LOG: background worker "logical replication launcher" (PID 4906) exited with exit code 1
2026-06-20 08:16:42.180 UTC [4901] LOG: shutting down
2026-06-20 08:16:42.364 UTC [4897] LOG: database system is shut down
2026-06-20T08:16:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50801 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:16:42 dist-test-slave-65w8 krb5kdc[6400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:16:42 dist-test-slave-65w8 krb5kdc[6400](info): set up 2 sockets
Jun 20 08:16:42 dist-test-slave-65w8 krb5kdc[6400](info): commencing operation
krb5kdc: starting...
W20260620 08:16:43.546120 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:16:43 dist-test-slave-65w8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:16:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:16:43.669013 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42445
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:42445,127.24.249.189:33795,127.24.249.188:45415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:43.750578 6413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:43.750847 6413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:43.750886 6413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:43.752086 6413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:43.752167 6413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:43.752192 6413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:43.752213 6413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:43.752242 6413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:43.753793 6413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:42445,127.24.249.189:33795,127.24.249.188:45415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:43.754114 6413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:43.754357 6413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:43.759673 6421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:43.759902 6413 server_base.cc:1034] running on GCE node
W20260620 08:16:43.759842 6419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:43.759842 6418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:43.760488 6413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:43.760826 6413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:43.761967 6413 hybrid_clock.cc:648] HybridClock initialized: now 1781943403761937 us; error 43 us; skew 500 ppm
Jun 20 08:16:43 dist-test-slave-65w8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:43.768837 6413 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:16:43.769793 6413 webserver.cc:460] Webserver started at http://127.24.249.190:39289/ using document root <none> and password file <none>
I20260620 08:16:43.770136 6413 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:43.770203 6413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:43.770354 6413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:43.771481 6413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "19bc56f0749e4aea97d5a2bfc491944a"
format_stamp: "Formatted at 2026-06-20 08:16:43 on dist-test-slave-65w8"
server_key: "b99e02e1fc7f8c9d0eeb5b0c8808f4fb"
server_key_iv: "e2d2ec0a3484fb4b34a96f429f69b508"
server_key_version: "encryptionkey@0"
I20260620 08:16:43.771870 6413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "19bc56f0749e4aea97d5a2bfc491944a"
format_stamp: "Formatted at 2026-06-20 08:16:43 on dist-test-slave-65w8"
server_key: "b99e02e1fc7f8c9d0eeb5b0c8808f4fb"
server_key_iv: "e2d2ec0a3484fb4b34a96f429f69b508"
server_key_version: "encryptionkey@0"
I20260620 08:16:43.773406 6413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:16:43.774485 6428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:43.774677 6413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:16:43.774775 6413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "19bc56f0749e4aea97d5a2bfc491944a"
format_stamp: "Formatted at 2026-06-20 08:16:43 on dist-test-slave-65w8"
server_key: "b99e02e1fc7f8c9d0eeb5b0c8808f4fb"
server_key_iv: "e2d2ec0a3484fb4b34a96f429f69b508"
server_key_version: "encryptionkey@0"
I20260620 08:16:43.774832 6413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:43.795236 6413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:43.797740 6413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:43.797928 6413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:43.801771 6413 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:42445
I20260620 08:16:43.801847 6480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:42445 every 8 connection(s)
I20260620 08:16:43.802204 6413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:16:43.803054 6481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:43.804579 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6413
I20260620 08:16:43.804693 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:16:43.804873 25574 external_mini_cluster.cc:1428] Setting key 93b428cbd655a6b724c17126a222ded1
I20260620 08:16:43.804706 6481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:16:43 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.189@KRBTEST.COM" created.
I20260620 08:16:43.815826 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.805531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42445 (local address 127.0.0.1:35014)
0620 08:16:43.805788 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.805834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:43.806563 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:43.806885 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:43.806887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:43.807145 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:43.807301 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:43.807305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.808899 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.808899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:43.809643 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:43.809645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.810416 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.810417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:43.810477 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:43.811018 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:43.811026 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:43.812336 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:43.814295 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:43.814301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:43.814303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:43.814502 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:43.814791 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:43.814792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:43.814792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:43.814910 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:43.815261 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:43.815268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:43.815358 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 08:16:43.815434 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:16:43.815615 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":157,"threads_started":1}
I20260620 08:16:43.815836 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.805563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.24.249.190:42445)
0620 08:16:43.805850 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:43.805854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:43.805944 (+ 90us) server_negotiation.cc:407] Connection header received
0620 08:16:43.806742 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:43.806745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:43.806776 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:43.806887 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:43.807405 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.808805 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.809734 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.810322 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.810451 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:43.812429 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:43.812439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:43.812439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:43.812446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:43.814219 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:43.814576 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:43.814578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:43.814603 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:43.814679 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:43.814977 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:43.814978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:43.814978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:43.815188 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:43.815255 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:43.815488 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:16:43.815715 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":264,"thread_start_us":144,"threads_started":1}
I20260620 08:16:43.816781 6481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:16:43.817159 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.817033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35346)
0620 08:16:43.817089 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.817138 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 08:16:43.817260 6429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
I20260620 08:16:43.817687 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.817548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35352)
0620 08:16:43.817621 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.817656 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 08:16:43.817839 6481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:33795: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab.
Entry for principal kudu/127.24.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab.
I20260620 08:16:43.831244 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:33795
--webserver_interface=127.24.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab
--principal=kudu/127.24.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:42445,127.24.249.189:33795,127.24.249.188:45415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:16:43.867105 6481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } attempt: 1
I20260620 08:16:43.867686 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.867529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35356)
0620 08:16:43.867599 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.867649 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:16:43.868161 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.868026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35362)
0620 08:16:43.868079 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.868117 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:16:43.868432 6481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:33795: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
W20260620 08:16:43.915117 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:43.915359 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:43.915383 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:43.916682 6488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:43.916728 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:43.916742 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:43.916755 6488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:43.916767 6488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:43.918342 6488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:42445,127.24.249.189:33795,127.24.249.188:45415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab
--principal=kudu/127.24.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:33795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:43.918632 6488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:43.918874 6488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:43.923904 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:43.923864 6496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:43.923924 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:43.924165 6488 server_base.cc:1034] running on GCE node
I20260620 08:16:43.924450 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:43.924790 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:43.925928 6488 hybrid_clock.cc:648] HybridClock initialized: now 1781943403925898 us; error 37 us; skew 500 ppm
Jun 20 08:16:43 dist-test-slave-65w8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:43.928416 6488 init.cc:377] Logged in from keytab as kudu/127.24.249.189@KRBTEST.COM (short username kudu)
I20260620 08:16:43.928871 6488 webserver.cc:460] Webserver started at http://127.24.249.189:43565/ using document root <none> and password file <none>
I20260620 08:16:43.929096 6488 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:43.929167 6488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:43.929314 6488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:43.930455 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data/instance:
uuid: "4b9257508b53465d81e49bf3c5f9b65c"
format_stamp: "Formatted at 2026-06-20 08:16:43 on dist-test-slave-65w8"
server_key: "ea3a8c928e2b3236d4dae80d1c64e5b8"
server_key_iv: "2d09ea414e6a42ac7b563e577450cb79"
server_key_version: "encryptionkey@0"
I20260620 08:16:43.930824 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/wal/instance:
uuid: "4b9257508b53465d81e49bf3c5f9b65c"
format_stamp: "Formatted at 2026-06-20 08:16:43 on dist-test-slave-65w8"
server_key: "ea3a8c928e2b3236d4dae80d1c64e5b8"
server_key_iv: "2d09ea414e6a42ac7b563e577450cb79"
server_key_version: "encryptionkey@0"
I20260620 08:16:43.932192 6488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:16:43.933004 6503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:43.933239 6488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:43.933357 6488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/wal
uuid: "4b9257508b53465d81e49bf3c5f9b65c"
format_stamp: "Formatted at 2026-06-20 08:16:43 on dist-test-slave-65w8"
server_key: "ea3a8c928e2b3236d4dae80d1c64e5b8"
server_key_iv: "2d09ea414e6a42ac7b563e577450cb79"
server_key_version: "encryptionkey@0"
I20260620 08:16:43.933436 6488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:43.936645 6481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } attempt: 2
I20260620 08:16:43.937083 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.936934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35368)
0620 08:16:43.937009 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.937050 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 08:16:43.937469 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.937359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35370)
0620 08:16:43.937410 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.937441 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:16:43.937558 6481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:33795: Network error: Client connection negotiation failed: client connection to 127.24.249.189:33795: connect: Connection refused (error 111)
I20260620 08:16:43.953431 6488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:43.955495 6488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:43.955674 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:43.960181 6488 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.189:33795
I20260620 08:16:43.960229 6556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.189:33795 every 8 connection(s)
I20260620 08:16:43.960623 6488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
I20260620 08:16:43.961478 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:43.963101 6557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:16:43.966885 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6488
I20260620 08:16:43.967010 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-1/wal/instance
I20260620 08:16:43.967218 25574 external_mini_cluster.cc:1428] Setting key c010a6b8a401181cfef0c227364ecf92
Jun 20 08:16:43 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:43.973203 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.963906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.24.249.190:42445)
0620 08:16:43.964005 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:43.964008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:43.964178 (+ 170us) server_negotiation.cc:407] Connection header received
0620 08:16:43.964631 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:43.964632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:43.964643 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:43.964700 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:43.965341 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.965852 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.967043 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.967389 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.967481 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:43.969897 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:43.969898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:43.969899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:43.969907 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:43.971450 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:43.971998 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:43.971999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:43.972000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:43.972069 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:43.972354 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:43.972355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:43.972355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:43.972501 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:43.972588 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:43.972894 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 08:16:43.973101 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 08:16:43.973497 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.963850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42445 (local address 127.0.0.1:35028)
0620 08:16:43.964036 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.964072 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:43.964525 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:43.964779 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:43.964782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:43.965030 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:43.965243 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:43.965247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.966032 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.966034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:43.966904 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:43.966907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.967504 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.967506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:43.967652 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:43.968241 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:43.968249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:43.969760 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:43.971609 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:43.971617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:43.971620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:43.971886 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:43.972148 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:43.972149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:43.972149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:43.972276 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:43.972581 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:43.972589 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:43.972748 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 08:16:43.972874 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 08:16:43.973268 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260620 08:16:43.974165 6557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.249.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.188@KRBTEST.COM" created.
Jun 20 08:16:43 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:16:43.982722 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.974457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35376)
0620 08:16:43.974511 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.974524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:43.974565 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:43.974879 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:43.974880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:43.975048 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:43.975099 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:43.975101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.976181 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.976182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:43.976746 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:43.976747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.977328 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.977329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:43.977401 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:43.977991 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:43.977997 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:43.979509 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:43.981548 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:43.981549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:43.981550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:43.981721 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:43.982015 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:43.982016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:43.982017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:43.982071 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:43.982380 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:43.982380 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:43.982419 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:16:43.982492 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:16:43.982598 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:16:43.982810 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.974554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.24.249.189:33795)
0620 08:16:43.974736 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:43.974741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:43.974758 (+ 17us) server_negotiation.cc:407] Connection header received
0620 08:16:43.974776 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:43.974777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:43.974788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:43.974879 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:43.975183 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.976064 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.976846 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:43.977232 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:43.977318 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:43.979641 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:43.979651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:43.979651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:43.979659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:43.981377 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:43.981820 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:43.981823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:43.981823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:43.981926 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:43.982157 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:43.982158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:43.982158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:43.982308 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:43.982402 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:43.982521 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 08:16:43.982672 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260620 08:16:43.983820 6557 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:16:43.984282 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.984121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53362)
0620 08:16:43.984182 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.984242 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 08:16:43.984455 6505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
I20260620 08:16:43.984946 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:43.984835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53374)
0620 08:16:43.984886 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:43.984920 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:16:43.985110 6557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45415: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab.
Entry for principal kudu/127.24.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab.
I20260620 08:16:43.995021 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:45415
--webserver_interface=127.24.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab
--principal=kudu/127.24.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:42445,127.24.249.189:33795,127.24.249.188:45415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:16:44.028759 6481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } attempt: 3
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:16:44.034379 6557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } attempt: 1
I20260620 08:16:44.035406 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.034704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53382)
0620 08:16:44.035346 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.035374 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 08:16:44.036028 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.035864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53398)
0620 08:16:44.035957 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.035991 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 08:16:44.036230 6557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45415: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
I20260620 08:16:44.036672 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.029171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35390)
0620 08:16:44.029246 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.029260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:44.029344 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:44.029501 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:44.029502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:44.029678 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:44.029751 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.029753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.030494 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.030496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.031008 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.031010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.031409 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.031410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.031493 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.031993 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:44.031998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:44.033349 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:44.035351 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.035353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.035353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.035593 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.035847 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.035849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.035849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.035997 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.036400 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:44.036400 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:44.036448 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 08:16:44.036514 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:16:44.036607 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 08:16:44.036806 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.029266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.24.249.189:33795)
0620 08:16:44.029342 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.029345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.029357 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:16:44.029410 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.029411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.029423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.029496 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.029842 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.030399 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.031107 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.031330 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.031411 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.033473 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.033475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.033476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.033483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.035259 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.035684 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.035686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.035687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.035753 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.036089 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.036090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.036091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.036289 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.036371 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.036573 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.036685 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:16:44.037259 6481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:16:44.037626 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.037518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53400)
0620 08:16:44.037568 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.037603 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 08:16:44.037964 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.037896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53412)
0620 08:16:44.037923 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.037945 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 08:16:44.038066 6481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45415: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
I20260620 08:16:44.069250 6481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } attempt: 1
I20260620 08:16:44.069949 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.069758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53420)
0620 08:16:44.069849 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.069905 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 08:16:44.070391 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.070293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53426)
0620 08:16:44.070342 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.070370 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:16:44.070526 6481 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45415: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
W20260620 08:16:44.084729 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:44.084991 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:44.085028 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:44.086304 6564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:44.086370 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:44.086396 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:44.086427 6564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:44.086448 6564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:44.087984 6564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:42445,127.24.249.189:33795,127.24.249.188:45415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab
--principal=kudu/127.24.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:45415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:44.088277 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:44.088529 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:44.093644 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:44.093647 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:44.093677 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:44.093783 6564 server_base.cc:1034] running on GCE node
I20260620 08:16:44.094165 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:44.094521 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:44.095726 6564 hybrid_clock.cc:648] HybridClock initialized: now 1781943404095694 us; error 63 us; skew 500 ppm
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:44.098683 6564 init.cc:377] Logged in from keytab as kudu/127.24.249.188@KRBTEST.COM (short username kudu)
I20260620 08:16:44.099217 6564 webserver.cc:460] Webserver started at http://127.24.249.188:32799/ using document root <none> and password file <none>
I20260620 08:16:44.099471 6564 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:44.099541 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:44.099694 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:44.100841 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data/instance:
uuid: "1ea610dd03114afe89246d5868627019"
format_stamp: "Formatted at 2026-06-20 08:16:44 on dist-test-slave-65w8"
server_key: "c9d8f45ba66fc182f92f211b07f28044"
server_key_iv: "3eae0efcce68ce4f6a3fdf827771d4bf"
server_key_version: "encryptionkey@0"
I20260620 08:16:44.101225 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/wal/instance:
uuid: "1ea610dd03114afe89246d5868627019"
format_stamp: "Formatted at 2026-06-20 08:16:44 on dist-test-slave-65w8"
server_key: "c9d8f45ba66fc182f92f211b07f28044"
server_key_iv: "3eae0efcce68ce4f6a3fdf827771d4bf"
server_key_version: "encryptionkey@0"
I20260620 08:16:44.102756 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:16:44.103670 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:44.103936 6564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:44.104061 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/wal
uuid: "1ea610dd03114afe89246d5868627019"
format_stamp: "Formatted at 2026-06-20 08:16:44 on dist-test-slave-65w8"
server_key: "c9d8f45ba66fc182f92f211b07f28044"
server_key_iv: "3eae0efcce68ce4f6a3fdf827771d4bf"
server_key_version: "encryptionkey@0"
I20260620 08:16:44.104144 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:44.104377 6557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } attempt: 2
I20260620 08:16:44.104794 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.104660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53434)
0620 08:16:44.104716 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.104749 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 08:16:44.105144 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.105037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53436)
0620 08:16:44.105080 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.105127 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 08:16:44.105229 6557 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45415: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45415: connect: Connection refused (error 111)
I20260620 08:16:44.118464 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:44.120890 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:44.121014 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:44.124809 6564 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.188:45415
I20260620 08:16:44.124895 6631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.188:45415 every 8 connection(s)
I20260620 08:16:44.125231 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
I20260620 08:16:44.126264 6632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:44.127822 6632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:16:44.130620 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6564
I20260620 08:16:44.130741 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781943243671421-25574-0/minicluster-data/master-2/wal/instance
I20260620 08:16:44.130934 25574 external_mini_cluster.cc:1428] Setting key e3f2de718c45eba8d3050b312dd8aa6e
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:16:44.138715 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.128547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.24.249.190:42445)
0620 08:16:44.128623 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.128625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.128795 (+ 170us) server_negotiation.cc:407] Connection header received
0620 08:16:44.129238 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.129240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.129249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.129310 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.130003 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.130504 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.131902 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.132156 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.132216 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.134823 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.134825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.134825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.134834 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.136786 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.137312 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.137313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.137314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.137373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.137723 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.137724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.137724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.137933 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.138030 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.138486 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.138584 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:16:44.138803 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.128495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42445 (local address 127.0.0.1:35040)
0620 08:16:44.128666 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.128709 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:44.129156 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:44.129384 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:44.129388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:44.129668 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:44.129916 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.129928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.130625 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.130626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.131782 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.131785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.132256 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.132258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.132404 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.132976 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:44.132985 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:44.134723 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:44.136888 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.136897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.136899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.137223 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.137450 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.137451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.137451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.137610 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.138036 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:44.138045 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:44.138222 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 08:16:44.138377 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 08:16:44.138543 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260620 08:16:44.139438 6632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:16:44.140878 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.131342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.24.249.188:45415)
0620 08:16:44.131502 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.131506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.131547 (+ 41us) server_negotiation.cc:407] Connection header received
0620 08:16:44.131709 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.131715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.131746 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.131820 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.132486 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.133191 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.134230 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.134670 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.134756 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.137573 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.137578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.137579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.137585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.139651 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.140031 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.140034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.140034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.140099 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.140312 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.140313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.140313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.140461 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.140638 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.140681 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.140763 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:16:44.145695 6481 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } attempt: 2
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:44.147231 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.139750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:33795 (local address 127.0.0.1:35404)
0620 08:16:44.139784 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.139795 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:44.139828 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:44.139975 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:44.139975 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:44.140103 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:44.140136 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.140137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.140768 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.140769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.141473 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.141475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.141959 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.141960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.142023 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.142469 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:44.142474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:44.143996 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:44.145977 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.145979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.145979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.146279 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.146524 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.146525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.146525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.146583 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.146863 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:44.146864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:44.146908 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 08:16:44.147007 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 08:16:44.147106 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 08:16:44.147234 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.139826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.24.249.189:33795)
0620 08:16:44.139869 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.139871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.139878 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:16:44.139901 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.139902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.139911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.139964 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.140213 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.140689 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.141562 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.141846 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.141960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.144096 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.144098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.144098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.144106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.145837 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.146366 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.146367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.146368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.146431 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.146668 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.146669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.146669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.146787 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.146860 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.147049 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.147139 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:16:44.147631 6632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:16:44.150759 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.141714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.24.249.190:42445)
0620 08:16:44.141758 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.141760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.141768 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:16:44.141790 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.141791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.141800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.141843 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.142449 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.143235 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.144150 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.144463 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.144599 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.147014 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.147015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.147015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.147022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.148904 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.149362 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.149363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.149364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.149426 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.149656 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.149657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.149657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.149770 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.150525 (+ 755us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.150565 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.150643 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943404, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:16:44.154721 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.146074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.24.249.188:45415)
0620 08:16:44.146134 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.146136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.146144 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:16:44.146164 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.146165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.146177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.146237 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.146603 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.147578 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.148540 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.148834 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.148909 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.150869 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.150870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.150870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.150876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.153475 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.153893 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.153895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.153895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.153950 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.154171 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.154172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.154173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.154295 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.154380 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.154548 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.154631 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:16:44.154719 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.146003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53448)
0620 08:16:44.146058 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.146069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:44.146106 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:44.146239 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:44.146240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:44.146417 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:44.146458 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.146460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.147656 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.147657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.148459 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.148462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.148911 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.148912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.148974 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.149389 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:44.149394 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:44.150776 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:44.153559 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.153560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.153560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.153785 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.154033 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.154034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.154034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.154095 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.154379 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:44.154380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:44.154430 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 08:16:44.154509 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 08:16:44.154613 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:16:44.155773 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.147912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53464)
0620 08:16:44.147951 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.147960 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:44.147995 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:44.148464 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:44.148465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:44.148626 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:44.148653 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.148655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.149444 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.149445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.150242 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.150245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.150745 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.150747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.150808 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.151199 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:44.151203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:44.152590 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:44.154578 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.154579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.154580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.154785 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.154992 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.154993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.154993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.155057 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.155383 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:44.155384 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:44.155432 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 08:16:44.155521 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 08:16:44.155629 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 08:16:44.155795 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.147942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.24.249.188:45415)
0620 08:16:44.148340 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.148342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.148349 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:16:44.148374 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.148375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.148383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.148463 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.148754 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.149375 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.150330 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.150661 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.150732 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.152675 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.152676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.152677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.152684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.154501 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.154865 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.154866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.154866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.154912 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.155117 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.155118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.155118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.155303 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.155375 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.155550 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.155674 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":378,"thread_start_us":343,"threads_started":1}
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:16:44.157191 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a: Bootstrap starting.
I20260620 08:16:44.157310 6632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: Bootstrap starting.
I20260620 08:16:44.158015 6481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:44.158041 6632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:44.158298 6481 log.cc:824] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:44.158361 6632 log.cc:824] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:44.159127 6481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a: No bootstrap required, opened a new log
I20260620 08:16:44.159242 6632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: No bootstrap required, opened a new log
I20260620 08:16:44.160313 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.151397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.24.249.189:33795)
0620 08:16:44.151442 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.151451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.151459 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:16:44.151480 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.151481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.151489 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.151543 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.152194 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.153267 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.154255 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.154552 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.154625 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.157381 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.157383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.157383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.157390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.159112 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.159512 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.159514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.159514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.159575 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.159780 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.159781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.159781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.159895 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.160083 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.160120 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.160199 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:16:44.161423 6632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.161589 6632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:44.161628 6632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea610dd03114afe89246d5868627019, State: Initialized, Role: FOLLOWER
I20260620 08:16:44.161671 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:16:44.161746 6632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.161757 6481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.161902 6481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:44.161945 6481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bc56f0749e4aea97d5a2bfc491944a, State: Initialized, Role: FOLLOWER
I20260620 08:16:44.162050 6481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.162092 6632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:44.162058 6640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.162143 6640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:16:44.162312 6641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.162352 6481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:44.162405 6641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:16:44.165371 6564 master_runner.cc:186] Error getting master registration for 127.24.249.190:42445: OK, instance_id { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" instance_seqno: 1781943403800702 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:16:44.167874 6564 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 08:16:44.164489 6663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:16:44.167955 6663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 08:16:44.171197 6672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:16:44.171357 6672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:16:44.168200 6413 master_runner.cc:186] Error getting master registration for 127.24.249.189:33795: OK, instance_id { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" instance_seqno: 1781943403958606 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:16:44.172480 6413 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:16:44.174525 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.164071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.24.249.189:33795)
0620 08:16:44.164107 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.164108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.164115 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:16:44.164133 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.164134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.164144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.164193 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.164956 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.165684 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.166779 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.167121 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.167191 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.169821 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.169823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.169824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.169831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.171748 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.172566 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.172568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.172568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.172624 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.173699 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.173700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.173700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.173836 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.173954 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.174344 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.174424 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:16:44.176020 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.163779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.24.249.190:42445)
0620 08:16:44.163907 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.163909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.163916 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:16:44.163938 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.163939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.163949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.163998 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.164971 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.165691 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.170458 (+ 4767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.170757 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.170827 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.172973 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.172974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.172975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.172981 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.174686 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.175137 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.175139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.175139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.175195 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.175401 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.175402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.175403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.175523 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.175631 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.175858 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.175933 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 08:16:44.176043 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.164151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.24.249.188:45415)
0620 08:16:44.164187 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.164189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.164195 (+ 6us) server_negotiation.cc:407] Connection header received
0620 08:16:44.164214 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.164215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.164224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.164270 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.167332 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.168174 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.169580 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.169940 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.169980 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.171761 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.171762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.171763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.171770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.173612 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.175172 (+ 1560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.175173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.175173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.175231 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.175438 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.175439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.175439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.175572 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.175663 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.175872 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.175956 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:16:44.196382 6557 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } attempt: 3
Jun 20 08:16:44 dist-test-slave-65w8 krb5kdc[6400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943403, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:16:44.205089 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.196696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45415 (local address 127.0.0.1:53484)
0620 08:16:44.196748 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:44.196759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:44.196797 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:44.196958 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:44.196959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:44.197157 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:44.197213 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.197215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.198167 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.198168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.199018 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:44.199021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.199495 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.199496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:44.199566 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.200480 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:44.200485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:44.202069 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:44.203970 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.203971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.203971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.204197 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.204412 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:44.204412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:44.204413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:44.204482 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:44.204763 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:44.204764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:44.204812 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 08:16:44.204884 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:16:44.204983 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 08:16:44.205101 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:44.196755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.24.249.188:45415)
0620 08:16:44.196797 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:44.196799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:44.196808 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:16:44.196862 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:44.196863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:44.196873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:44.196939 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:44.197302 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.198069 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.199099 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:44.199410 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:44.199484 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:44.202160 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:44.202161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:44.202162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:44.202169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:44.203880 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.204276 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.204277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.204277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.204338 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:44.204555 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:44.204556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:44.204556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:44.204692 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:44.204762 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:44.204922 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 08:16:44.205002 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:16:44.207341 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: Bootstrap starting.
I20260620 08:16:44.208107 6557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:44.208400 6557 log.cc:824] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:44.209302 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: No bootstrap required, opened a new log
I20260620 08:16:44.211321 6557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.211462 6557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:44.211495 6557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9257508b53465d81e49bf3c5f9b65c, State: Initialized, Role: FOLLOWER
I20260620 08:16:44.211674 6557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.212004 6557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:44.211968 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.212155 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:16:44.213927 6686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:16:44.214031 6686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:16:44.217427 6488 master_runner.cc:263] No leader master found from master 4b9257508b53465d81e49bf3c5f9b65c
I20260620 08:16:44.217482 6488 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 08:16:44.298767 6434 tablet.cc:2369] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:16:44.371605 6585 tablet.cc:2369] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:16:44.430215 6641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:44.430349 6641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.430703 6641 leader_election.cc:288] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b9257508b53465d81e49bf3c5f9b65c (127.24.249.189:33795), 1ea610dd03114afe89246d5868627019 (127.24.249.188:45415)
I20260620 08:16:44.431033 6606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19bc56f0749e4aea97d5a2bfc491944a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea610dd03114afe89246d5868627019" is_pre_election: true
I20260620 08:16:44.431046 6530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19bc56f0749e4aea97d5a2bfc491944a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9257508b53465d81e49bf3c5f9b65c" is_pre_election: true
I20260620 08:16:44.431218 6530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bc56f0749e4aea97d5a2bfc491944a in term 0.
I20260620 08:16:44.431216 6606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bc56f0749e4aea97d5a2bfc491944a in term 0.
I20260620 08:16:44.431427 6429 leader_election.cc:302] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bc56f0749e4aea97d5a2bfc491944a, 4b9257508b53465d81e49bf3c5f9b65c; no voters:
I20260620 08:16:44.431618 6641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:16:44.431679 6641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:16:44.431725 6641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:44.432463 6641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.432613 6641 leader_election.cc:288] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [CANDIDATE]: Term 1 election: Requested vote from peers 4b9257508b53465d81e49bf3c5f9b65c (127.24.249.189:33795), 1ea610dd03114afe89246d5868627019 (127.24.249.188:45415)
I20260620 08:16:44.432770 6530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19bc56f0749e4aea97d5a2bfc491944a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b9257508b53465d81e49bf3c5f9b65c"
I20260620 08:16:44.432775 6606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19bc56f0749e4aea97d5a2bfc491944a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea610dd03114afe89246d5868627019"
I20260620 08:16:44.432866 6606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:44.432865 6530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:44.433768 6606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bc56f0749e4aea97d5a2bfc491944a in term 1.
I20260620 08:16:44.433948 6530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bc56f0749e4aea97d5a2bfc491944a in term 1.
I20260620 08:16:44.433979 6430 leader_election.cc:302] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bc56f0749e4aea97d5a2bfc491944a, 1ea610dd03114afe89246d5868627019; no voters:
I20260620 08:16:44.434064 6641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:16:44.434211 6641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [term 1 LEADER]: Becoming Leader. State: Replica: 19bc56f0749e4aea97d5a2bfc491944a, State: Running, Role: LEADER
I20260620 08:16:44.434329 6641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } }
I20260620 08:16:44.434810 6688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19bc56f0749e4aea97d5a2bfc491944a. Latest consensus state: current_term: 1 leader_uuid: "19bc56f0749e4aea97d5a2bfc491944a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.434921 6688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:44.435148 6690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:16:44.435340 6690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:16:44.436947 6530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [term 1 FOLLOWER]: Refusing update from remote peer 19bc56f0749e4aea97d5a2bfc491944a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:44.437240 6606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [term 1 FOLLOWER]: Refusing update from remote peer 19bc56f0749e4aea97d5a2bfc491944a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:44.437273 6641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:16:44.437544 6641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:16:44.438565 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19bc56f0749e4aea97d5a2bfc491944a. Latest consensus state: current_term: 1 leader_uuid: "19bc56f0749e4aea97d5a2bfc491944a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.438679 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:16:44.438696 6640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19bc56f0749e4aea97d5a2bfc491944a. Latest consensus state: current_term: 1 leader_uuid: "19bc56f0749e4aea97d5a2bfc491944a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.438792 6640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:16:44.439842 6641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19bc56f0749e4aea97d5a2bfc491944a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.439937 6641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:44.439867 6688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19bc56f0749e4aea97d5a2bfc491944a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.439989 6688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:44.440018 6692 mvcc.cc:205] Tried to move back new op lower bound from 7298840184570982400 to 7298840184563691520. Current Snapshot: MvccSnapshot[applied={T|T < 7298840184570982400}]
I20260620 08:16:44.440086 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19bc56f0749e4aea97d5a2bfc491944a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.440068 6640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19bc56f0749e4aea97d5a2bfc491944a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19bc56f0749e4aea97d5a2bfc491944a" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42445 } } peers { permanent_uuid: "4b9257508b53465d81e49bf3c5f9b65c" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 33795 } } peers { permanent_uuid: "1ea610dd03114afe89246d5868627019" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45415 } } }
I20260620 08:16:44.440107 6693 mvcc.cc:205] Tried to move back new op lower bound from 7298840184570982400 to 7298840184563691520. Current Snapshot: MvccSnapshot[applied={T|T < 7298840184570982400}]
I20260620 08:16:44.440146 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:16:44.440146 6640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:16:44.440749 6690 catalog_manager.cc:1347] Generated new cluster ID: ed081be9e3694a23a09acdf8c7352df5
I20260620 08:16:44.440811 6690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:16:44.452515 6690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:16:44.453188 6690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:16:44.463732 6690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19bc56f0749e4aea97d5a2bfc491944a: Generated new TSK 0
I20260620 08:16:44.464012 6690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:16:45.168499 6663 catalog_manager.cc:1259] Loaded cluster ID: ed081be9e3694a23a09acdf8c7352df5
I20260620 08:16:45.168581 6663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: loading cluster ID for follower catalog manager: success
I20260620 08:16:45.169356 6663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: acquiring CA information for follower catalog manager: success
I20260620 08:16:45.169620 6663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ea610dd03114afe89246d5868627019: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:16:45.214545 6686 catalog_manager.cc:1259] Loaded cluster ID: ed081be9e3694a23a09acdf8c7352df5
I20260620 08:16:45.214697 6686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: loading cluster ID for follower catalog manager: success
I20260620 08:16:45.215518 6686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: acquiring CA information for follower catalog manager: success
I20260620 08:16:45.215873 6686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b9257508b53465d81e49bf3c5f9b65c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:16:45.530292 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6413
I20260620 08:16:45.535959 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6488
I20260620 08:16:45.541121 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6564
2026-06-20T08:16:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3060 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:16:45 dist-test-slave-65w8 krb5kdc[6705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:16:45 dist-test-slave-65w8 krb5kdc[6705](info): set up 2 sockets
Jun 20 08:16:45 dist-test-slave-65w8 krb5kdc[6705](info): commencing operation
krb5kdc: starting...
W20260620 08:16:46.590301 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:16:46 dist-test-slave-65w8 krb5kdc[6705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:16:46.717953 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43873
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:43873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:46.803423 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:46.803648 6718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:46.803673 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:46.804790 6718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:46.804827 6718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:46.804842 6718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:46.804855 6718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:46.804867 6718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:46.806352 6718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:43873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:46.806556 6718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:46.806741 6718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:46.811875 6723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:46.811877 6724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:46.812196 6726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:46.812165 6718 server_base.cc:1034] running on GCE node
I20260620 08:16:46.812497 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:46.812805 6718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:46.813968 6718 hybrid_clock.cc:648] HybridClock initialized: now 1781943406813931 us; error 52 us; skew 500 ppm
Jun 20 08:16:46 dist-test-slave-65w8 krb5kdc[6705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943406, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:46.816640 6718 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:16:46.817121 6718 webserver.cc:460] Webserver started at http://127.24.249.190:44875/ using document root <none> and password file <none>
I20260620 08:16:46.817351 6718 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:46.817420 6718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:46.817574 6718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:46.818707 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c"
format_stamp: "Formatted at 2026-06-20 08:16:46 on dist-test-slave-65w8"
server_key: "95116f4cccc377cf630a2015495eb718"
server_key_iv: "dfddca5a65f6d019152fc6c1d9076245"
server_key_version: "encryptionkey@0"
I20260620 08:16:46.819078 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c"
format_stamp: "Formatted at 2026-06-20 08:16:46 on dist-test-slave-65w8"
server_key: "95116f4cccc377cf630a2015495eb718"
server_key_iv: "dfddca5a65f6d019152fc6c1d9076245"
server_key_version: "encryptionkey@0"
I20260620 08:16:46.820698 6718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:16:46.821563 6733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:46.821950 6718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:16:46.822085 6718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c"
format_stamp: "Formatted at 2026-06-20 08:16:46 on dist-test-slave-65w8"
server_key: "95116f4cccc377cf630a2015495eb718"
server_key_iv: "dfddca5a65f6d019152fc6c1d9076245"
server_key_version: "encryptionkey@0"
I20260620 08:16:46.822170 6718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:46.847708 6718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:46.850391 6718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:46.850598 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:46.855376 6718 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:43873
I20260620 08:16:46.855429 6785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:43873 every 8 connection(s)
I20260620 08:16:46.855860 6718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:16:46.856778 6786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:46.859234 6786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c: Bootstrap starting.
I20260620 08:16:46.859833 6786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:46.860179 6786 log.cc:824] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:46.860983 6786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c: No bootstrap required, opened a new log
I20260620 08:16:46.862789 6786 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43873 } }
I20260620 08:16:46.862954 6786 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:46.862979 6786 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8d5c0c73f04992ac0ca612eeb3da7c, State: Initialized, Role: FOLLOWER
I20260620 08:16:46.863137 6786 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43873 } }
I20260620 08:16:46.863215 6786 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:16:46.863258 6786 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:16:46.863315 6786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:46.863754 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6718
I20260620 08:16:46.863848 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:16:46.863972 6786 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43873 } }
I20260620 08:16:46.864069 25574 external_mini_cluster.cc:1428] Setting key bf3b4566e6e95de549200a3f63749d32
I20260620 08:16:46.864102 6786 leader_election.cc:302] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8d5c0c73f04992ac0ca612eeb3da7c; no voters:
I20260620 08:16:46.864318 6786 leader_election.cc:288] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:16:46.864387 6789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:16:46.864542 6789 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [term 1 LEADER]: Becoming Leader. State: Replica: ae8d5c0c73f04992ac0ca612eeb3da7c, State: Running, Role: LEADER
I20260620 08:16:46.864612 6786 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:46.864681 6789 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43873 } }
I20260620 08:16:46.865293 6791 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8d5c0c73f04992ac0ca612eeb3da7c. Latest consensus state: current_term: 1 leader_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43873 } } }
I20260620 08:16:46.865583 6790 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8d5c0c73f04992ac0ca612eeb3da7c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43873 } } }
I20260620 08:16:46.865840 6791 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:46.865929 6790 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c [sys.catalog]: This master's current role is: LEADER
W20260620 08:16:46.866715 6807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:16:46.866808 6807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:16:46.866881 6806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:16:46.867081 6806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:16:46.868703 6806 catalog_manager.cc:1347] Generated new cluster ID: 355198f8626746b4aadf1d8477714f99
I20260620 08:16:46.868755 6806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:16:46 dist-test-slave-65w8 krb5kdc[6705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:46.878301 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:46.864833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.24.249.190:43873)
0620 08:16:46.865020 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:46.865031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:46.865355 (+ 324us) server_negotiation.cc:407] Connection header received
0620 08:16:46.865487 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:46.865496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:46.865571 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:46.865819 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:46.866826 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:46.868425 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:46.869543 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:46.870276 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:46.870394 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:46.873206 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:46.873210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:46.873211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:46.873224 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:46.876364 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:46.876832 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:46.876835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:46.876835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:46.876979 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:46.877218 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:46.877219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:46.877220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:46.877571 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:46.877758 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:46.877888 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 08:16:46.878051 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260620 08:16:46.879662 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:16:46.882030 6806 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:16:46.882592 6806 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:16:46.889555 6806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae8d5c0c73f04992ac0ca612eeb3da7c: Generated new TSK 0
I20260620 08:16:46.889775 6806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:16:46.934302 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6718
2026-06-20T08:16:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1392 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:16:46 dist-test-slave-65w8 krb5kdc[6813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:16:46 dist-test-slave-65w8 krb5kdc[6813](info): set up 2 sockets
Jun 20 08:16:46 dist-test-slave-65w8 krb5kdc[6813](info): commencing operation
krb5kdc: starting...
W20260620 08:16:47.982262 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:16:48.117100 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34027
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:48.205703 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:48.205973 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:48.206009 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:48.207176 6827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:48.207243 6827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:48.207271 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:48.207298 6827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:48.207321 6827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:48.208786 6827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:48.209064 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:48.209283 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:48.214366 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:48.214426 6827 server_base.cc:1034] running on GCE node
W20260620 08:16:48.214378 6832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:48.214390 6833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:48.214785 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:48.215082 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:48.216264 6827 hybrid_clock.cc:648] HybridClock initialized: now 1781943408216247 us; error 33 us; skew 500 ppm
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:48.218813 6827 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:16:48.219287 6827 webserver.cc:460] Webserver started at http://127.24.249.190:43159/ using document root <none> and password file <none>
I20260620 08:16:48.219518 6827 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:48.219594 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:48.219743 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:48.220811 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "5e7734d7d11c4c22b7ff74958fb0a081"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "434c75a05e5858c4f2d1daca81102c95"
server_key_iv: "fbd2ad4355481a6871a6fcdcac474ed5"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.221179 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "5e7734d7d11c4c22b7ff74958fb0a081"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "434c75a05e5858c4f2d1daca81102c95"
server_key_iv: "fbd2ad4355481a6871a6fcdcac474ed5"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.222679 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:16:48.223461 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.223687 6827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:48.223800 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "5e7734d7d11c4c22b7ff74958fb0a081"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "434c75a05e5858c4f2d1daca81102c95"
server_key_iv: "fbd2ad4355481a6871a6fcdcac474ed5"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.223877 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:48.233800 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:48.235872 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:48.236037 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:48.240373 6827 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:34027
I20260620 08:16:48.240408 6895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:34027 every 8 connection(s)
I20260620 08:16:48.240769 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:16:48.241791 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:48.243516 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6827
I20260620 08:16:48.243649 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:16:48.243849 25574 external_mini_cluster.cc:1428] Setting key 69665f8a747272eed8fbf0e0ab3a06bf
I20260620 08:16:48.245010 6896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Bootstrap starting.
I20260620 08:16:48.245739 6896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:48.246073 6896 log.cc:824] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:48.246863 6896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: No bootstrap required, opened a new log
I20260620 08:16:48.249040 6896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:48.249332 6896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:48.249379 6896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7734d7d11c4c22b7ff74958fb0a081, State: Initialized, Role: FOLLOWER
I20260620 08:16:48.249562 6896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:48.249673 6896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:16:48.249714 6896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:16:48.249791 6896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:48.250602 6896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:48.250732 6896 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7734d7d11c4c22b7ff74958fb0a081; no voters:
I20260620 08:16:48.250922 6896 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:16:48.250965 6901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:16:48.251188 6901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 LEADER]: Becoming Leader. State: Replica: 5e7734d7d11c4c22b7ff74958fb0a081, State: Running, Role: LEADER
I20260620 08:16:48.251232 6896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:48.251317 6901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:48.251726 6901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7734d7d11c4c22b7ff74958fb0a081. Latest consensus state: current_term: 1 leader_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } } }
I20260620 08:16:48.251879 6901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:48.252104 6910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:48.252740 6910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:16:48.252759 6902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } } }
I20260620 08:16:48.253336 6902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:48.254675 6910 catalog_manager.cc:1347] Generated new cluster ID: db9798b2eaab4ae5b4fb1635159708f4
I20260620 08:16:48.254741 6910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:16:48.256763 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.244405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.24.249.190:34027)
0620 08:16:48.244677 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.244687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.244720 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:16:48.245260 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.245270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.245334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.245507 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:48.246319 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.247697 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.248785 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.249501 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.249620 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.253061 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:48.253066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:48.253066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:48.253079 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:48.255180 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.255565 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.255568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.255569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.255662 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.255875 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.255877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.255877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.256130 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:48.256316 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.256399 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.256545 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":155,"threads_started":1}
I20260620 08:16:48.271171 6910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:16:48.271903 6910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:16:48.279614 6910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Generated new TSK 0
I20260620 08:16:48.279819 6910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:16:48.317826 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34027
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:48.419091 6922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:48.419299 6922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:48.419325 6922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:48.421103 6922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:48.421154 6922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:48.421182 6922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:16:48.422765 6922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:48.423040 6922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:48.423283 6922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:48.428856 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:48.428885 6931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:48.428952 6922 server_base.cc:1034] running on GCE node
W20260620 08:16:48.429193 6929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:48.429469 6922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:48.429790 6922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:48.430984 6922 hybrid_clock.cc:648] HybridClock initialized: now 1781943408430955 us; error 48 us; skew 500 ppm
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:48.434501 6922 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:16:48.435082 6922 webserver.cc:460] Webserver started at http://127.24.249.129:38117/ using document root <none> and password file <none>
I20260620 08:16:48.435371 6922 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:48.435439 6922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:48.435585 6922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:48.437522 6922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "d659d6399bb04810bcece137244d4bae"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "4e137d31f41d8947b331a8e43bd8186a"
server_key_iv: "1b91694b9a99d781b4a21fa9fbd14676"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.438340 6922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "d659d6399bb04810bcece137244d4bae"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "4e137d31f41d8947b331a8e43bd8186a"
server_key_iv: "1b91694b9a99d781b4a21fa9fbd14676"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.440804 6922 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 08:16:48.441818 6938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.442233 6922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:48.442338 6922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "d659d6399bb04810bcece137244d4bae"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "4e137d31f41d8947b331a8e43bd8186a"
server_key_iv: "1b91694b9a99d781b4a21fa9fbd14676"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.442440 6922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:48.470921 6922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:48.474112 6922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:48.474293 6922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:48.474598 6922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:48.474964 6922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:48.475031 6922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.475066 6922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:48.475090 6922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.481720 6922 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:38265
I20260620 08:16:48.481794 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:38265 every 8 connection(s)
I20260620 08:16:48.482144 6922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:16:48.484180 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 6922
I20260620 08:16:48.484304 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:16:48.484589 25574 external_mini_cluster.cc:1428] Setting key 6439571bde37a36d991b82ce11f23240
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:16:48.494053 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.483094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:44173 (local address 127.24.249.190:34027)
0620 08:16:48.483180 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.483182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.483367 (+ 185us) server_negotiation.cc:407] Connection header received
0620 08:16:48.483881 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.483882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.483895 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.484009 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:48.485311 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.486146 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.487563 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.487920 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.487988 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.490975 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:48.490976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:48.490977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:48.490985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:48.492497 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.492942 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.492943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.492944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.493010 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.493308 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.493309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.493310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.493436 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:48.493521 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.493804 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.493937 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:16:48.494273 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.483050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:44173)
0620 08:16:48.483219 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:48.483262 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:48.483766 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:48.484052 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:48.484055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:48.484872 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:48.485180 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.485187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.486248 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.486250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.487464 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.487468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.488018 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.488020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.488191 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.488892 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:48.488905 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:48.490868 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:48.492609 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:48.492619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:48.492622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:48.492861 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:48.493102 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:48.493103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:48.493103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:48.493227 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:48.493505 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:48.493512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:48.493668 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 08:16:48.493784 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 08:16:48.493983 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260620 08:16:48.495098 7052 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34027
I20260620 08:16:48.495198 7052 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:48.495348 7052 heartbeater.cc:511] Master 127.24.249.190:34027 requested a full tablet report, sending...
I20260620 08:16:48.495995 6860 ts_manager.cc:194] Registered new tserver with Master: d659d6399bb04810bcece137244d4bae (127.24.249.129:38265)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
I20260620 08:16:48.496757 6860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:44173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:16:48.513815 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34027
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:48.602007 7057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:48.602312 7057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:48.602365 7057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:48.603619 7057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:48.603691 7057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:48.603749 7057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:16:48.606657 7057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:48.606926 7057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:48.607213 7057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:48.614300 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:48.614306 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:48.614413 7057 server_base.cc:1034] running on GCE node
W20260620 08:16:48.614656 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:48.614933 7057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:48.615187 7057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:48.616376 7057 hybrid_clock.cc:648] HybridClock initialized: now 1781943408616364 us; error 27 us; skew 500 ppm
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:48.619016 7057 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:16:48.619469 7057 webserver.cc:460] Webserver started at http://127.24.249.130:46619/ using document root <none> and password file <none>
I20260620 08:16:48.619721 7057 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:48.619807 7057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:48.619925 7057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:48.620922 7057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "ce71b96b19cb43169a6eb9df8caadd03"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "6fdb98e371512982f920840f5ec00d1f"
server_key_iv: "0009ba0d0bb42be33c0aaba6d8188913"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.621218 7057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "ce71b96b19cb43169a6eb9df8caadd03"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "6fdb98e371512982f920840f5ec00d1f"
server_key_iv: "0009ba0d0bb42be33c0aaba6d8188913"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.622620 7057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:16:48.623359 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.623591 7057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:48.623669 7057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "ce71b96b19cb43169a6eb9df8caadd03"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "6fdb98e371512982f920840f5ec00d1f"
server_key_iv: "0009ba0d0bb42be33c0aaba6d8188913"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.623769 7057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:48.658697 7057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:48.661142 7057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:48.661294 7057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:48.661585 7057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:48.661964 7057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:48.662003 7057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.662026 7057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:48.662039 7057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.668044 7057 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:43899
I20260620 08:16:48.668097 7186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:43899 every 8 connection(s)
I20260620 08:16:48.668403 7057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:16:48.669695 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7057
I20260620 08:16:48.669803 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:16:48.670029 25574 external_mini_cluster.cc:1428] Setting key 45f1b2c95b7b03a8d30aae2574ea2735
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:16:48.679706 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.669104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:59157 (local address 127.24.249.190:34027)
0620 08:16:48.669177 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.669179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.669384 (+ 205us) server_negotiation.cc:407] Connection header received
0620 08:16:48.669983 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.669984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.670000 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.670063 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:48.671235 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.672001 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.673476 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.673708 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.673776 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.676429 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:48.676431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:48.676431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:48.676438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:48.678010 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.678502 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.678503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.678503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.678592 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.678901 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.678902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.678902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.679035 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:48.679142 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.679516 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.679610 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 08:16:48.679852 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.669080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:59157)
0620 08:16:48.669255 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:48.669296 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:48.669834 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:48.670201 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:48.670205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:48.670882 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:48.671148 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.671154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.672112 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.672113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.673386 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.673389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.673801 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.673802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.673973 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.674506 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:48.674517 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:48.676319 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:48.678090 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:48.678098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:48.678100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:48.678421 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:48.678669 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:48.678670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:48.678671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:48.678811 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:48.679177 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:48.679184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:48.679332 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 08:16:48.679446 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 08:16:48.679619 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260620 08:16:48.680408 7187 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34027
I20260620 08:16:48.680508 7187 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:48.680661 7187 heartbeater.cc:511] Master 127.24.249.190:34027 requested a full tablet report, sending...
I20260620 08:16:48.681083 6860 ts_manager.cc:194] Registered new tserver with Master: ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
I20260620 08:16:48.681587 6860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:59157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:16:48.701404 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34027
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:48.786507 7193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:48.786746 7193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:48.786787 7193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:48.787949 7193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:48.788013 7193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:48.788059 7193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:16:48.789548 7193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:48.789825 7193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:48.790097 7193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:48.795094 7201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:48.795113 7199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:48.795439 7198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:48.795596 7193 server_base.cc:1034] running on GCE node
I20260620 08:16:48.795835 7193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:48.796095 7193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:48.797339 7193 hybrid_clock.cc:648] HybridClock initialized: now 1781943408797312 us; error 39 us; skew 500 ppm
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:48.805712 7193 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:16:48.806219 7193 webserver.cc:460] Webserver started at http://127.24.249.131:33159/ using document root <none> and password file <none>
I20260620 08:16:48.806473 7193 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:48.806517 7193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:48.806680 7193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:16:48.807752 7193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "21ad019cd89245a5aca2712263d43240"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "bf939c6c784120d0bbabbea5b1a9671f"
server_key_iv: "32ae72d43c384ce4a66256324931225a"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.808101 7193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "21ad019cd89245a5aca2712263d43240"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "bf939c6c784120d0bbabbea5b1a9671f"
server_key_iv: "32ae72d43c384ce4a66256324931225a"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.809514 7193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:16:48.810415 7208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.810621 7193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:16:48.810716 7193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "21ad019cd89245a5aca2712263d43240"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "bf939c6c784120d0bbabbea5b1a9671f"
server_key_iv: "32ae72d43c384ce4a66256324931225a"
server_key_version: "encryptionkey@0"
I20260620 08:16:48.810796 7193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:48.827739 7193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:48.830240 7193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:48.830392 7193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:48.830641 7193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:48.830958 7193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:48.830993 7193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.831017 7193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:48.831063 7193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:48.837466 7193 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:36497
I20260620 08:16:48.837512 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:36497 every 8 connection(s)
I20260620 08:16:48.837818 7193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:16:48 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943408, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:48.847211 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7193
I20260620 08:16:48.847303 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:16:48.847496 25574 external_mini_cluster.cc:1428] Setting key 95b9b646526b0afa9181948f9b834d35
I20260620 08:16:48.847903 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.838677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:51005 (local address 127.24.249.190:34027)
0620 08:16:48.838725 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.838727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.838865 (+ 138us) server_negotiation.cc:407] Connection header received
0620 08:16:48.839370 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.839371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.839382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.839439 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:48.840356 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.840888 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.841896 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.842170 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.842239 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.844510 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:48.844512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:48.844512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:48.844527 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:48.846070 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.846501 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.846502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.846502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.846596 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.846998 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.847000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.847000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.847155 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:48.847215 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.847660 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.847779 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:16:48.848269 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.838560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:51005)
0620 08:16:48.838724 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:48.838765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:48.839258 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:48.839500 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:48.839503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:48.840057 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:48.840249 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.840253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.841001 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.841002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.841780 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.841782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.842279 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.842281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.842401 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.842894 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:48.842902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:48.844365 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:48.846149 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:48.846156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:48.846158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:48.846401 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:48.846733 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:48.846734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:48.846734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:48.846874 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:48.847235 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:48.847243 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:48.847439 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 08:16:48.847574 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 08:16:48.847771 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 08:16:48.848978 7322 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34027
I20260620 08:16:48.849073 7322 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:48.849236 7322 heartbeater.cc:511] Master 127.24.249.190:34027 requested a full tablet report, sending...
I20260620 08:16:48.849578 6860 ts_manager.cc:194] Registered new tserver with Master: 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497)
I20260620 08:16:48.850083 6860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:51005
I20260620 08:16:48.860026 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:16:48.868577 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.861505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.24.249.190:34027)
0620 08:16:48.861589 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.861592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.861727 (+ 135us) server_negotiation.cc:407] Connection header received
0620 08:16:48.861831 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.861832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.861843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.861939 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:48.862528 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.863025 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.863850 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.864063 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.864142 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.865383 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:48.865385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:48.865385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:48.865393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:48.866934 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.867427 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.867428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.867429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.867545 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:48.867831 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:48.867832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:48.867832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:48.868156 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:48.868293 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.868384 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.868460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 08:16:48.870604 6860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:16:48.872378 6860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:16:48.879261 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.874961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38265 (local address 127.0.0.1:33186)
0620 08:16:48.875611 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:48.875621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:48.875670 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:48.875941 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:48.875942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:48.875944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:48.875988 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.875990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.876831 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.876831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.878075 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.878078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.878963 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.878964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.879041 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.879057 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 08:16:48.879123 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 08:16:48.879182 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":115,"threads_started":1}
I20260620 08:16:48.879438 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.875238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.24.249.131:36497)
0620 08:16:48.875447 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.875451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.875460 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:16:48.875500 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.875502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.875518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.875620 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:48.875835 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.877103 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.878346 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.879042 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.879109 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.879155 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.879292 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.879334 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
I20260620 08:16:48.879580 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.874957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:36497 (local address 127.0.0.1:40468)
0620 08:16:48.875360 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:48.875368 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:48.875408 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:48.875645 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:48.875648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:48.875650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:48.875727 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.875728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.877218 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.877219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.878269 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.878271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.879109 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.879110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.879181 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.879185 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:16:48.879273 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:16:48.879428 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":134,"threads_started":1}
I20260620 08:16:48.880450 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.874940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:43899 (local address 127.0.0.1:33992)
0620 08:16:48.875142 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:48.875184 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:48.875251 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:48.875714 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:48.875715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:48.875717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:48.875779 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.875781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.878337 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.878337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.879041 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:48.879042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.880160 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.880161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:48.880243 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.880247 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:16:48.880335 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:16:48.880381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260620 08:16:48.880479 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.875123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.24.249.130:43899)
0620 08:16:48.875312 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.875318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.875329 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:16:48.875377 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.875379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.875399 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.875497 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:48.875870 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.877100 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.879144 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.879950 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.880035 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.880112 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.880356 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.880400 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260620 08:16:48.880409 7256 tablet_service.cc:1467] Processing CreateTablet for tablet a726b611d95a4edca53278949d764c2a (DEFAULT_TABLE table=test-table [id=f9760d3194bd40c8b734192e27a85c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:48.880818 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726b611d95a4edca53278949d764c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:48.881343 7121 tablet_service.cc:1467] Processing CreateTablet for tablet a726b611d95a4edca53278949d764c2a (DEFAULT_TABLE table=test-table [id=f9760d3194bd40c8b734192e27a85c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:48.881631 7121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726b611d95a4edca53278949d764c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:48.881919 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:48.875340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.24.249.129:38265)
0620 08:16:48.875545 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:48.875548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:48.875726 (+ 178us) server_negotiation.cc:407] Connection header received
0620 08:16:48.875770 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:48.875772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:48.875788 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:48.875858 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:48.876046 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.876728 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.878153 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:48.878871 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:48.878954 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:48.879016 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:48.879757 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 08:16:48.879805 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
I20260620 08:16:48.883913 7339 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Bootstrap starting.
I20260620 08:16:48.884652 7339 tablet_bootstrap.cc:654] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:48.884524 6986 tablet_service.cc:1467] Processing CreateTablet for tablet a726b611d95a4edca53278949d764c2a (DEFAULT_TABLE table=test-table [id=f9760d3194bd40c8b734192e27a85c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:48.884886 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726b611d95a4edca53278949d764c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:48.885229 7339 log.cc:824] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:48.886029 7340 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Bootstrap starting.
I20260620 08:16:48.886076 7339 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: No bootstrap required, opened a new log
I20260620 08:16:48.886137 7339 ts_tablet_manager.cc:1397] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:16:48.886595 7340 tablet_bootstrap.cc:654] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:48.887068 7340 log.cc:824] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:48.887746 7340 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: No bootstrap required, opened a new log
I20260620 08:16:48.887825 7340 ts_tablet_manager.cc:1397] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:16:48.888332 7343 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Bootstrap starting.
I20260620 08:16:48.888530 7339 raft_consensus.cc:348] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:48.888657 7339 raft_consensus.cc:374] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:48.888684 7339 raft_consensus.cc:729] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ad019cd89245a5aca2712263d43240, State: Initialized, Role: FOLLOWER
I20260620 08:16:48.888799 7339 consensus_queue.cc:260] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:48.889052 7322 heartbeater.cc:503] Master 127.24.249.190:34027 was elected leader, sending a full tablet report...
I20260620 08:16:48.889070 7339 ts_tablet_manager.cc:1428] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:16:48.889261 7343 tablet_bootstrap.cc:654] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:48.889832 7343 log.cc:824] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:48.889999 7340 raft_consensus.cc:348] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:48.890190 7340 raft_consensus.cc:374] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:48.890234 7340 raft_consensus.cc:729] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce71b96b19cb43169a6eb9df8caadd03, State: Initialized, Role: FOLLOWER
I20260620 08:16:48.890362 7340 consensus_queue.cc:260] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:48.890599 7343 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: No bootstrap required, opened a new log
I20260620 08:16:48.890661 7343 ts_tablet_manager.cc:1397] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:16:48.890653 7340 ts_tablet_manager.cc:1428] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:16:48.890681 7187 heartbeater.cc:503] Master 127.24.249.190:34027 was elected leader, sending a full tablet report...
I20260620 08:16:48.892263 7343 raft_consensus.cc:348] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:48.892385 7343 raft_consensus.cc:374] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:48.892405 7343 raft_consensus.cc:729] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d659d6399bb04810bcece137244d4bae, State: Initialized, Role: FOLLOWER
I20260620 08:16:48.892531 7343 consensus_queue.cc:260] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:48.892802 7343 ts_tablet_manager.cc:1428] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:16:48.892848 7052 heartbeater.cc:503] Master 127.24.249.190:34027 was elected leader, sending a full tablet report...
W20260620 08:16:48.919054 7188 tablet.cc:2369] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:16:48.983091 7053 tablet.cc:2369] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:16:49.080883 7344 raft_consensus.cc:482] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:49.081020 7344 raft_consensus.cc:504] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.081525 7344 leader_election.cc:288] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899), d659d6399bb04810bcece137244d4bae (127.24.249.129:38265)
I20260620 08:16:49.084813 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.081757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38265 (local address 127.24.249.131:53799)
0620 08:16:49.081819 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.081831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.081876 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.082096 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.082097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.082099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:49.082202 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.082205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.082943 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.082944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.083782 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.083784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.084539 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.084540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.084623 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.084627 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.084690 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.084732 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":18}
I20260620 08:16:49.084825 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.081818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:53799 (local address 127.24.249.129:38265)
0620 08:16:49.081954 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.081957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.081970 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:16:49.082004 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.082006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.082021 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.082087 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:49.082303 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.082848 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.083884 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.084450 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.084515 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.084523 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.084722 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.084761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 08:16:49.085245 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.081984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:60587 (local address 127.24.249.130:43899)
0620 08:16:49.082065 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.082066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.082167 (+ 101us) server_negotiation.cc:407] Connection header received
0620 08:16:49.082224 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.082226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.082239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.082291 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:49.082494 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.083038 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.084270 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.084824 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.084896 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.084905 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.085138 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.085182 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:16:49.085243 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.081855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:43899 (local address 127.24.249.131:60587)
0620 08:16:49.082092 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.082103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.082166 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.082307 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.082307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.082309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:49.082375 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.082376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.083142 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.083143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.084170 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.084173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.084922 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.084923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.085014 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.085019 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.085124 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.085166 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":149,"threads_started":1}
I20260620 08:16:49.085204 7006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726b611d95a4edca53278949d764c2a" candidate_uuid: "21ad019cd89245a5aca2712263d43240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d659d6399bb04810bcece137244d4bae" is_pre_election: true
I20260620 08:16:49.085392 7006 raft_consensus.cc:2454] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21ad019cd89245a5aca2712263d43240 in term 0.
I20260620 08:16:49.085642 7210 leader_election.cc:302] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ad019cd89245a5aca2712263d43240, d659d6399bb04810bcece137244d4bae; no voters:
I20260620 08:16:49.085619 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726b611d95a4edca53278949d764c2a" candidate_uuid: "21ad019cd89245a5aca2712263d43240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce71b96b19cb43169a6eb9df8caadd03" is_pre_election: true
I20260620 08:16:49.085872 7141 raft_consensus.cc:2454] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21ad019cd89245a5aca2712263d43240 in term 0.
I20260620 08:16:49.085892 7344 raft_consensus.cc:2781] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:16:49.085950 7344 raft_consensus.cc:482] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:16:49.086002 7344 raft_consensus.cc:3037] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:49.086767 7344 raft_consensus.cc:504] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.086936 7344 leader_election.cc:288] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [CANDIDATE]: Term 1 election: Requested vote from peers ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899), d659d6399bb04810bcece137244d4bae (127.24.249.129:38265)
I20260620 08:16:49.087108 7141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726b611d95a4edca53278949d764c2a" candidate_uuid: "21ad019cd89245a5aca2712263d43240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce71b96b19cb43169a6eb9df8caadd03"
I20260620 08:16:49.087106 7006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726b611d95a4edca53278949d764c2a" candidate_uuid: "21ad019cd89245a5aca2712263d43240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d659d6399bb04810bcece137244d4bae"
I20260620 08:16:49.087193 7006 raft_consensus.cc:3037] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:49.087210 7141 raft_consensus.cc:3037] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:49.087911 7006 raft_consensus.cc:2454] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21ad019cd89245a5aca2712263d43240 in term 1.
I20260620 08:16:49.088035 7141 raft_consensus.cc:2454] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21ad019cd89245a5aca2712263d43240 in term 1.
I20260620 08:16:49.088078 7210 leader_election.cc:302] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ad019cd89245a5aca2712263d43240, d659d6399bb04810bcece137244d4bae; no voters:
I20260620 08:16:49.088171 7344 raft_consensus.cc:2781] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:16:49.088353 7344 raft_consensus.cc:686] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 1 LEADER]: Becoming Leader. State: Replica: 21ad019cd89245a5aca2712263d43240, State: Running, Role: LEADER
W20260620 08:16:49.088366 7323 tablet.cc:2369] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:16:49.088471 7344 consensus_queue.cc:237] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.089385 6860 catalog_manager.cc:5495] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21ad019cd89245a5aca2712263d43240 (127.24.249.131). New cstate: current_term: 1 leader_uuid: "21ad019cd89245a5aca2712263d43240" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } health_report { overall_health: UNKNOWN } } }
I20260620 08:16:49.107551 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.104731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.24.249.131:36497)
0620 08:16:49.104797 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.104799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.104807 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:16:49.104879 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.104879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.104899 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.104967 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:16:49.105187 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.105714 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.106623 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.106838 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.106909 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.107121 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:16:49.107227 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:16:49.107296 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.107375 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.107404 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:16:49.109534 7006 raft_consensus.cc:1261] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Refusing update from remote peer 21ad019cd89245a5aca2712263d43240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:49.109509 7141 raft_consensus.cc:1261] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Refusing update from remote peer 21ad019cd89245a5aca2712263d43240: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:49.109946 7344 consensus_queue.cc:1035] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:16:49.110106 7349 consensus_queue.cc:1035] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:16:49.113255 7352 mvcc.cc:205] Tried to move back new op lower bound from 7298840203709505536 to 7298840203626864640. Current Snapshot: MvccSnapshot[applied={T|T < 7298840203709505536}]
I20260620 08:16:49.113185 7354 mvcc.cc:205] Tried to move back new op lower bound from 7298840203709505536 to 7298840203626864640. Current Snapshot: MvccSnapshot[applied={T|T < 7298840203709505536}]
I20260620 08:16:49.113970 7353 mvcc.cc:205] Tried to move back new op lower bound from 7298840203709505536 to 7298840203626864640. Current Snapshot: MvccSnapshot[applied={T|T < 7298840203709505536}]
I20260620 08:16:49.116325 6860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:16:49.116436 6860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:16:49.117404 6860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Sending DeleteTablet for 3 replicas of tablet a726b611d95a4edca53278949d764c2a
I20260620 08:16:49.117821 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet a726b611d95a4edca53278949d764c2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:49 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:40468
I20260620 08:16:49.117827 7121 tablet_service.cc:1514] Processing DeleteTablet for tablet a726b611d95a4edca53278949d764c2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:49 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:33992
I20260620 08:16:49.118270 6986 tablet_service.cc:1514] Processing DeleteTablet for tablet a726b611d95a4edca53278949d764c2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:49 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:33186
I20260620 08:16:49.118336 7362 tablet_replica.cc:330] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: stopping tablet replica
I20260620 08:16:49.118392 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6922
I20260620 08:16:49.120613 7362 raft_consensus.cc:2229] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:16:49.120786 7362 raft_consensus.cc:2258] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:49.121117 7361 tablet_replica.cc:330] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: stopping tablet replica
I20260620 08:16:49.121244 7361 raft_consensus.cc:2229] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:16:49.121398 7361 raft_consensus.cc:2258] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:49.121677 7362 ts_tablet_manager.cc:1905] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:16:49.121853 7361 ts_tablet_manager.cc:1905] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:16:49.123924 7361 ts_tablet_manager.cc:1918] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:16:49.123924 7362 ts_tablet_manager.cc:1918] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:16:49.124017 7361 log.cc:1196] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/a726b611d95a4edca53278949d764c2a
I20260620 08:16:49.124182 7362 log.cc:1196] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/a726b611d95a4edca53278949d764c2a
I20260620 08:16:49.124341 7361 ts_tablet_manager.cc:1939] T a726b611d95a4edca53278949d764c2a P 21ad019cd89245a5aca2712263d43240: Deleting consensus metadata
I20260620 08:16:49.124603 7362 ts_tablet_manager.cc:1939] T a726b611d95a4edca53278949d764c2a P ce71b96b19cb43169a6eb9df8caadd03: Deleting consensus metadata
I20260620 08:16:49.124918 6847 catalog_manager.cc:4841] TS 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497): tablet a726b611d95a4edca53278949d764c2a (table test-table [id=f9760d3194bd40c8b734192e27a85c9d]) successfully deleted
I20260620 08:16:49.125097 6844 catalog_manager.cc:4841] TS ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899): tablet a726b611d95a4edca53278949d764c2a (table test-table [id=f9760d3194bd40c8b734192e27a85c9d]) successfully deleted
W20260620 08:16:49.125277 6845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:38265 (error 108)
I20260620 08:16:49.125837 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.125689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38265 (local address 127.0.0.1:33194)
0620 08:16:49.125749 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.125794 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 08:16:49.125903 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 7057
W20260620 08:16:49.130548 6845 catalog_manager.cc:4584] TS d659d6399bb04810bcece137244d4bae (127.24.249.129:38265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a726b611d95a4edca53278949d764c2a: Network error: Client connection negotiation failed: client connection to 127.24.249.129:38265: connect: Connection refused (error 111)
I20260620 08:16:49.131811 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 7193
I20260620 08:16:49.136638 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 6827
I20260620 08:16:49.142201 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34027
--webserver_interface=127.24.249.190
--webserver_port=43159
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:49.227458 7365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:49.227710 7365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:49.227748 7365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:49.228905 7365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:49.228972 7365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:49.228996 7365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:49.229033 7365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:49.229058 7365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:49.230563 7365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:49.230850 7365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:49.231079 7365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:49.242151 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.242147 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.242147 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:49.242426 7365 server_base.cc:1034] running on GCE node
I20260620 08:16:49.242842 7365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:49.243214 7365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:49.244431 7365 hybrid_clock.cc:648] HybridClock initialized: now 1781943409244398 us; error 53 us; skew 500 ppm
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:49.247339 7365 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:16:49.247844 7365 webserver.cc:460] Webserver started at http://127.24.249.190:43159/ using document root <none> and password file <none>
I20260620 08:16:49.248102 7365 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:49.248189 7365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:49.249662 7365 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:16:49.250583 7381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.250792 7365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:16:49.250918 7365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "5e7734d7d11c4c22b7ff74958fb0a081"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "434c75a05e5858c4f2d1daca81102c95"
server_key_iv: "fbd2ad4355481a6871a6fcdcac474ed5"
server_key_version: "encryptionkey@0"
I20260620 08:16:49.251369 7365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:49.262120 7365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:49.264340 7365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:49.264603 7365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:49.269070 7365 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:34027
I20260620 08:16:49.269120 7434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:34027 every 8 connection(s)
I20260620 08:16:49.269569 7365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:16:49.271271 7435 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:16:49.271947 7435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Bootstrap starting.
I20260620 08:16:49.277719 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7365
I20260620 08:16:49.278124 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:38265
--local_ip_for_outbound_sockets=127.24.249.129
--tserver_master_addrs=127.24.249.190:34027
--webserver_port=38117
--webserver_interface=127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:16:49.281250 7435 log.cc:824] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:49.283377 7435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:16:49.283779 7435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Bootstrap complete.
I20260620 08:16:49.286201 7435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.286513 7435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7734d7d11c4c22b7ff74958fb0a081, State: Initialized, Role: FOLLOWER
I20260620 08:16:49.286718 7435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.286811 7435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:16:49.286868 7435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:16:49.286917 7435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:16:49.287916 7435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.288033 7435 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7734d7d11c4c22b7ff74958fb0a081; no voters:
I20260620 08:16:49.288215 7435 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:16:49.288250 7439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:16:49.288357 7439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 LEADER]: Becoming Leader. State: Replica: 5e7734d7d11c4c22b7ff74958fb0a081, State: Running, Role: LEADER
I20260620 08:16:49.288441 7439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.288496 7435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:49.288710 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7734d7d11c4c22b7ff74958fb0a081. Latest consensus state: current_term: 2 leader_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } } }
I20260620 08:16:49.288787 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:49.288715 7440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } } }
I20260620 08:16:49.288951 7440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:49.288965 7443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:16:49.289844 7443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:16:49.290028 7443 catalog_manager.cc:1259] Loaded cluster ID: db9798b2eaab4ae5b4fb1635159708f4
I20260620 08:16:49.290053 7443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:16:49.290272 7443 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 08:16:49.295554 7443 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 08:16:49.298177 7443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:16:49.298373 7443 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Loaded TSK: 0
I20260620 08:16:49.298594 7443 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:16:49.364970 7437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:49.365212 7437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:49.365239 7437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:49.366601 7437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:49.366649 7437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:49.366678 7437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:16:49.368203 7437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:38265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=38117
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:49.368460 7437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:49.368718 7437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:49.373749 7463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.373878 7462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.373759 7465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:49.374141 7437 server_base.cc:1034] running on GCE node
I20260620 08:16:49.374344 7437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:49.374609 7437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:49.375793 7437 hybrid_clock.cc:648] HybridClock initialized: now 1781943409375781 us; error 31 us; skew 500 ppm
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:49.378765 7437 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:16:49.379300 7437 webserver.cc:460] Webserver started at http://127.24.249.129:38117/ using document root <none> and password file <none>
I20260620 08:16:49.379554 7437 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:49.379612 7437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:49.381006 7437 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:16:49.381732 7472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.381938 7437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:49.382040 7437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "d659d6399bb04810bcece137244d4bae"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "4e137d31f41d8947b331a8e43bd8186a"
server_key_iv: "1b91694b9a99d781b4a21fa9fbd14676"
server_key_version: "encryptionkey@0"
I20260620 08:16:49.382397 7437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:49.411005 7437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:49.413569 7437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:49.413756 7437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:49.414062 7437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:49.414525 7480 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:16:49.416110 7437 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:16:49.416204 7437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 08:16:49.416253 7437 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:16:49.416973 7437 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:16:49.417038 7437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:16:49.417147 7480 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Bootstrap starting.
I20260620 08:16:49.424611 7437 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:38265
I20260620 08:16:49.424705 7587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:38265 every 8 connection(s)
I20260620 08:16:49.425032 7437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:16:49.426978 7480 log.cc:824] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:49.429046 7480 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:16:49.429342 7480 tablet_bootstrap.cc:492] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Bootstrap complete.
I20260620 08:16:49.430346 7480 ts_tablet_manager.cc:1397] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.013s
I20260620 08:16:49.431344 7480 raft_consensus.cc:348] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.431596 7480 raft_consensus.cc:729] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d659d6399bb04810bcece137244d4bae, State: Initialized, Role: FOLLOWER
I20260620 08:16:49.431774 7480 consensus_queue.cc:260] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.432060 7480 ts_tablet_manager.cc:1428] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:49.434355 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7437
I20260620 08:16:49.434661 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:43899
--local_ip_for_outbound_sockets=127.24.249.130
--tserver_master_addrs=127.24.249.190:34027
--webserver_port=46619
--webserver_interface=127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:16:49.437083 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.425925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:56495 (local address 127.24.249.190:34027)
0620 08:16:49.426090 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.426099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.426137 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:16:49.426960 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.426966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.427012 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.427182 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:49.427872 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.428764 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.429967 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.430365 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.430497 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.432830 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:49.432834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:49.432834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:49.432846 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:49.434869 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.435328 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.435331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.435332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.435458 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.435784 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.435785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.435786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.436262 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:49.436377 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.436798 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.436921 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260620 08:16:49.437183 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.425733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:56495)
0620 08:16:49.425965 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.426024 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.426722 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.427238 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.427242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.427529 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:49.427737 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.427744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.428887 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.428888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.429826 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.429829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.430490 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.430492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.430638 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.431201 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:49.431209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:49.432715 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:49.434960 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.434968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.434970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.435235 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.435564 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.435565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.435566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.435700 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.436392 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:49.436401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:49.436581 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.436720 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.436934 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260620 08:16:49.438256 7588 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34027
I20260620 08:16:49.438371 7588 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:49.438575 7588 heartbeater.cc:511] Master 127.24.249.190:34027 requested a full tablet report, sending...
I20260620 08:16:49.439407 7399 ts_manager.cc:194] Registered new tserver with Master: d659d6399bb04810bcece137244d4bae (127.24.249.129:38265)
I20260620 08:16:49.440816 7399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:56495
I20260620 08:16:49.441850 7588 heartbeater.cc:503] Master 127.24.249.190:34027 was elected leader, sending a full tablet report...
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:16:49.449518 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.440262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.24.249.129:38265)
0620 08:16:49.440432 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.440437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.440652 (+ 215us) server_negotiation.cc:407] Connection header received
0620 08:16:49.440783 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.440785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.440800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.440896 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:49.441554 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.442758 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.443605 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.444150 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.444234 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.446310 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:49.446319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:49.446320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:49.446329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:49.448106 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.448585 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.448588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.448588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.448653 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.448845 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.448846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.448847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.449022 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:49.449114 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.449311 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.449414 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260620 08:16:49.449527 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.440204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38265 (local address 127.0.0.1:33204)
0620 08:16:49.440538 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.440566 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.440675 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.440994 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.441000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.441387 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:49.441451 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.441462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.442867 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.442868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.443471 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.443473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.444245 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.444246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.444309 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.444899 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:49.444903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:49.446222 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:49.448296 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.448303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.448304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.448499 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.448720 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.448721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.448721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.448784 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.449126 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:49.449133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:49.449179 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.449259 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.449434 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":111,"threads_started":1}
I20260620 08:16:49.450280 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet a726b611d95a4edca53278949d764c2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:49 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:33204
I20260620 08:16:49.450866 7597 tablet_replica.cc:330] stopping tablet replica
I20260620 08:16:49.450980 7597 raft_consensus.cc:2229] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:16:49.451082 7597 raft_consensus.cc:2258] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:49.451403 7597 ts_tablet_manager.cc:1905] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:16:49.453316 7597 ts_tablet_manager.cc:1918] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:16:49.453380 7597 log.cc:1196] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/a726b611d95a4edca53278949d764c2a
I20260620 08:16:49.453619 7597 ts_tablet_manager.cc:1939] T a726b611d95a4edca53278949d764c2a P d659d6399bb04810bcece137244d4bae: Deleting consensus metadata
I20260620 08:16:49.454154 7383 catalog_manager.cc:4841] TS d659d6399bb04810bcece137244d4bae (127.24.249.129:38265): tablet a726b611d95a4edca53278949d764c2a (table test-table [id=f9760d3194bd40c8b734192e27a85c9d]) successfully deleted
W20260620 08:16:49.521365 7594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:49.521616 7594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:49.521659 7594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:49.522922 7594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:49.522993 7594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:49.523039 7594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:16:49.524550 7594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:43899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=46619
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:49.524824 7594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:49.525054 7594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:49.530061 7602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.530056 7605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.530059 7603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:49.530366 7594 server_base.cc:1034] running on GCE node
I20260620 08:16:49.530582 7594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:49.530827 7594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:49.532011 7594 hybrid_clock.cc:648] HybridClock initialized: now 1781943409531997 us; error 27 us; skew 500 ppm
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:49.534837 7594 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:16:49.535362 7594 webserver.cc:460] Webserver started at http://127.24.249.130:46619/ using document root <none> and password file <none>
I20260620 08:16:49.535595 7594 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:49.535684 7594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:49.537046 7594 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:16:49.537671 7612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.537935 7594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:16:49.538041 7594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "ce71b96b19cb43169a6eb9df8caadd03"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "6fdb98e371512982f920840f5ec00d1f"
server_key_iv: "0009ba0d0bb42be33c0aaba6d8188913"
server_key_version: "encryptionkey@0"
I20260620 08:16:49.538404 7594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:49.557036 7594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:49.559248 7594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:49.559418 7594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:49.559649 7594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:49.560003 7594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:49.560058 7594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.560108 7594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:49.560137 7594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.566675 7594 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:43899
I20260620 08:16:49.566720 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:43899 every 8 connection(s)
I20260620 08:16:49.567113 7594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:16:49.571271 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7594
I20260620 08:16:49.571710 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:36497
--local_ip_for_outbound_sockets=127.24.249.131
--tserver_master_addrs=127.24.249.190:34027
--webserver_port=33159
--webserver_interface=127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:49.577322 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.567931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:41095 (local address 127.24.249.190:34027)
0620 08:16:49.567977 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.567979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.568272 (+ 293us) server_negotiation.cc:407] Connection header received
0620 08:16:49.568805 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.568807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.568821 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.568875 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:49.569805 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.570331 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.571388 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.571663 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.571757 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.574293 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:49.574294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:49.574294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:49.574304 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:49.575865 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.576296 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.576298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.576298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.576363 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.576639 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.576640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.576640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.576767 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:49.576848 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.577131 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.577230 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:16:49.577562 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.567929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:41095)
0620 08:16:49.568107 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.568144 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.568695 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.568927 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.568931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.569497 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:49.569693 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.569698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.570436 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.570437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.571268 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.571270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.571771 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.571772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.571927 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.572563 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:49.572571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:49.574176 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:49.575963 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.575971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.575973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.576213 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.576437 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.576438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.576438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.576559 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.576839 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:49.576846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:49.576994 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.577111 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.577274 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260620 08:16:49.578085 7726 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34027
I20260620 08:16:49.578184 7726 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:49.578384 7726 heartbeater.cc:511] Master 127.24.249.190:34027 requested a full tablet report, sending...
I20260620 08:16:49.578823 7399 ts_manager.cc:194] Registered new tserver with Master: ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
I20260620 08:16:49.579384 7399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:41095
W20260620 08:16:49.654774 7729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:49.654979 7729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:49.655002 7729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:49.656198 7729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:49.656241 7729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:49.656268 7729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:16:49.657627 7729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:36497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=33159
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:49.657807 7729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:49.658064 7729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:49.663537 7734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:49.663688 7729 server_base.cc:1034] running on GCE node
W20260620 08:16:49.663532 7735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.663532 7737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:49.663967 7729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:49.664237 7729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:49.665410 7729 hybrid_clock.cc:648] HybridClock initialized: now 1781943409665388 us; error 44 us; skew 500 ppm
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:49.667994 7729 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:16:49.668426 7729 webserver.cc:460] Webserver started at http://127.24.249.131:33159/ using document root <none> and password file <none>
I20260620 08:16:49.668658 7729 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:49.668715 7729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:49.670198 7729 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:49.670861 7744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.671057 7729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:49.671160 7729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "21ad019cd89245a5aca2712263d43240"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "bf939c6c784120d0bbabbea5b1a9671f"
server_key_iv: "32ae72d43c384ce4a66256324931225a"
server_key_version: "encryptionkey@0"
I20260620 08:16:49.671532 7729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:49.692090 7729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:49.694350 7729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:49.694501 7729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:49.694769 7729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:49.695087 7729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:49.695143 7729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.695185 7729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:49.695214 7729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.701092 7729 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:36497
I20260620 08:16:49.701143 7857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:36497 every 8 connection(s)
I20260620 08:16:49.701483 7729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:16:49.707810 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7729
I20260620 08:16:49.712219 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.702219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:53679 (local address 127.24.249.190:34027)
0620 08:16:49.702265 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.702268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.702503 (+ 235us) server_negotiation.cc:407] Connection header received
0620 08:16:49.703042 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.703043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.703057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.703101 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:49.704082 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.704587 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.705609 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.705812 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.705928 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.708502 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:49.708504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:49.708505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:49.708512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:49.710247 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.710726 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.710727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.710728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.710795 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.711267 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.711270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.711271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.711494 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:49.711587 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.712010 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.712109 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:16:49.712316 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.702213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:53679)
0620 08:16:49.702379 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.702414 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.702933 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.703171 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.703176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.703767 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:49.703958 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.703964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.704686 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.704687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.705498 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.705500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.705930 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.705931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.706093 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.706650 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:49.706658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:49.708386 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:49.710353 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.710361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.710363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.710647 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.710891 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:49.710892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:49.710892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:49.711169 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:49.711606 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:49.711622 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:49.711816 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.711944 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.712113 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260620 08:16:49.712890 7858 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34027
I20260620 08:16:49.713054 7858 heartbeater.cc:464] Registering TS with master...
I20260620 08:16:49.713250 7858 heartbeater.cc:511] Master 127.24.249.190:34027 requested a full tablet report, sending...
I20260620 08:16:49.713701 7399 ts_manager.cc:194] Registered new tserver with Master: 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497)
I20260620 08:16:49.714105 7399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:53679
I20260620 08:16:49.715828 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.708266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.24.249.190:34027)
0620 08:16:49.708408 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.708410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.708421 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:16:49.708501 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.708502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.708522 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.708585 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:49.709291 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.710016 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.711043 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.711343 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.711395 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.712943 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:49.712944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:49.712944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:49.712951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:49.714776 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.715139 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.715140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.715140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.715194 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:49.715394 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:49.715395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:49.715395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:49.715487 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:49.715618 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.715674 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.715737 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260620 08:16:49.716570 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:16:49.719543 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.716971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.24.249.190:34027)
0620 08:16:49.717054 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.717056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.717120 (+ 64us) server_negotiation.cc:407] Connection header received
0620 08:16:49.717200 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.717200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.717248 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.717329 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:16:49.717477 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.717993 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.718758 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.718971 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.719050 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.719232 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:16:49.719333 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:16:49.719433 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.719450 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.719473 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 08:16:49.719879 7399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:16:49.721591 7399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:16:49.724174 7522 tablet_service.cc:1467] Processing CreateTablet for tablet 879a4f1bc525407ab56965ff9604723f (DEFAULT_TABLE table=test-table [id=49266f35b63f4cc9983de5326965b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:49.724385 7522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879a4f1bc525407ab56965ff9604723f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:49.726290 7480 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Bootstrap starting.
I20260620 08:16:49.726622 7480 tablet_bootstrap.cc:654] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:49.727716 7480 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: No bootstrap required, opened a new log
I20260620 08:16:49.727777 7480 ts_tablet_manager.cc:1397] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:16:49.727926 7480 raft_consensus.cc:348] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.728006 7480 raft_consensus.cc:374] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:49.728062 7480 raft_consensus.cc:729] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d659d6399bb04810bcece137244d4bae, State: Initialized, Role: FOLLOWER
I20260620 08:16:49.728111 7480 consensus_queue.cc:260] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.728217 7480 ts_tablet_manager.cc:1428] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 08:16:49.728860 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.723929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:43899 (local address 127.0.0.1:33996)
0620 08:16:49.724081 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.724093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.724143 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.724577 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.724578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.724581 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:49.724640 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.724642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.726085 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.726085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.727493 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.727497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.728564 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.728565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.728634 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.728662 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.728736 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.728783 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":53,"thread_start_us":106,"threads_started":1}
I20260620 08:16:49.728874 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.724202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.24.249.130:43899)
0620 08:16:49.724383 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.724388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.724400 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:16:49.724445 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.724448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.724465 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.724557 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:49.724713 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.725980 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.727586 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.728417 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.728502 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.728567 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.728754 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.728796 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260620 08:16:49.729331 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.723863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:36497 (local address 127.0.0.1:40484)
0620 08:16:49.723913 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.723924 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.723963 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.724601 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.724601 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.724603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:49.724649 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.724650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.726042 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.726043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.728107 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.728110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.729083 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.729084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.729154 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.729159 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.729223 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.729268 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":990}
I20260620 08:16:49.729393 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.724051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.24.249.131:36497)
0620 08:16:49.724433 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.724437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.724445 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:16:49.724481 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.724483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.724499 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.724719 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:49.724792 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.725916 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.728208 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.728971 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.729089 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.729147 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.729236 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.729280 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":94,"threads_started":1}
I20260620 08:16:49.729611 7660 tablet_service.cc:1467] Processing CreateTablet for tablet 879a4f1bc525407ab56965ff9604723f (DEFAULT_TABLE table=test-table [id=49266f35b63f4cc9983de5326965b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:49.730018 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879a4f1bc525407ab56965ff9604723f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:49.730098 7792 tablet_service.cc:1467] Processing CreateTablet for tablet 879a4f1bc525407ab56965ff9604723f (DEFAULT_TABLE table=test-table [id=49266f35b63f4cc9983de5326965b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:16:49.730422 7792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879a4f1bc525407ab56965ff9604723f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:16:49.732751 7868 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Bootstrap starting.
I20260620 08:16:49.733124 7869 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Bootstrap starting.
I20260620 08:16:49.733568 7868 tablet_bootstrap.cc:654] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:49.733704 7869 tablet_bootstrap.cc:654] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Neither blocks nor log segments found. Creating new log.
I20260620 08:16:49.734153 7868 log.cc:824] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:49.734262 7869 log.cc:824] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:49.734920 7868 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: No bootstrap required, opened a new log
I20260620 08:16:49.734977 7868 ts_tablet_manager.cc:1397] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:16:49.734974 7869 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: No bootstrap required, opened a new log
I20260620 08:16:49.735035 7869 ts_tablet_manager.cc:1397] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:16:49.736555 7868 raft_consensus.cc:348] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.736713 7868 raft_consensus.cc:374] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:49.736761 7868 raft_consensus.cc:729] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce71b96b19cb43169a6eb9df8caadd03, State: Initialized, Role: FOLLOWER
I20260620 08:16:49.736665 7869 raft_consensus.cc:348] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.736836 7869 raft_consensus.cc:374] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:16:49.736881 7869 raft_consensus.cc:729] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ad019cd89245a5aca2712263d43240, State: Initialized, Role: FOLLOWER
I20260620 08:16:49.736886 7868 consensus_queue.cc:260] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.736984 7869 consensus_queue.cc:260] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.737213 7868 ts_tablet_manager.cc:1428] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:16:49.737275 7726 heartbeater.cc:503] Master 127.24.249.190:34027 was elected leader, sending a full tablet report...
I20260620 08:16:49.737298 7869 ts_tablet_manager.cc:1428] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:16:49.737423 7858 heartbeater.cc:503] Master 127.24.249.190:34027 was elected leader, sending a full tablet report...
I20260620 08:16:49.765206 7593 raft_consensus.cc:482] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:49.765417 7593 raft_consensus.cc:504] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.765848 7593 leader_election.cc:288] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497), ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
I20260620 08:16:49.768736 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.766035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:36497 (local address 127.24.249.129:44693)
0620 08:16:49.766093 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.766107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.766166 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.766358 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.766360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.766362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:49.766440 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.766443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.767128 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.767130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.767877 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.767879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.768436 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.768437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.768523 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.768550 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.768605 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.768639 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":9}
I20260620 08:16:49.768772 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.766108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:44693 (local address 127.24.249.131:36497)
0620 08:16:49.766168 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.766171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.766180 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:16:49.766272 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.766273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.766284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.766342 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:49.766517 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.767047 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.767964 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.768354 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.768415 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.768422 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.768646 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.768696 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:16:49.768823 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.766194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:33395 (local address 127.24.249.130:43899)
0620 08:16:49.766238 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.766240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.766310 (+ 70us) server_negotiation.cc:407] Connection header received
0620 08:16:49.766373 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.766374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.766385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.766444 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:16:49.766626 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.767073 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.768028 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.768422 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.768495 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.768504 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.768707 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.768750 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:16:49.769048 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.766104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:43899 (local address 127.24.249.129:33395)
0620 08:16:49.766253 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.766263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:49.766324 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:49.766456 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:49.766457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:49.766458 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:16:49.766552 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.766554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.767182 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.767183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.767932 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:49.767934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.768516 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.768517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:49.768617 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.768622 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:16:49.768683 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:16:49.768718 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260620 08:16:49.769148 7812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a4f1bc525407ab56965ff9604723f" candidate_uuid: "d659d6399bb04810bcece137244d4bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ad019cd89245a5aca2712263d43240" is_pre_election: true
I20260620 08:16:49.769341 7812 raft_consensus.cc:2454] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d659d6399bb04810bcece137244d4bae in term 0.
I20260620 08:16:49.769402 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a4f1bc525407ab56965ff9604723f" candidate_uuid: "d659d6399bb04810bcece137244d4bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce71b96b19cb43169a6eb9df8caadd03" is_pre_election: true
I20260620 08:16:49.769577 7680 raft_consensus.cc:2454] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d659d6399bb04810bcece137244d4bae in term 0.
I20260620 08:16:49.769611 7476 leader_election.cc:302] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ad019cd89245a5aca2712263d43240, d659d6399bb04810bcece137244d4bae; no voters:
I20260620 08:16:49.769727 7593 raft_consensus.cc:2781] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:16:49.769785 7593 raft_consensus.cc:482] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:16:49.769819 7593 raft_consensus.cc:3037] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:49.770531 7593 raft_consensus.cc:504] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.770695 7593 leader_election.cc:288] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 1 election: Requested vote from peers 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497), ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
I20260620 08:16:49.770838 7812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a4f1bc525407ab56965ff9604723f" candidate_uuid: "d659d6399bb04810bcece137244d4bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ad019cd89245a5aca2712263d43240"
I20260620 08:16:49.770843 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a4f1bc525407ab56965ff9604723f" candidate_uuid: "d659d6399bb04810bcece137244d4bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce71b96b19cb43169a6eb9df8caadd03"
I20260620 08:16:49.770915 7812 raft_consensus.cc:3037] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:49.770915 7680 raft_consensus.cc:3037] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:16:49.771807 7680 raft_consensus.cc:2454] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d659d6399bb04810bcece137244d4bae in term 1.
I20260620 08:16:49.771807 7812 raft_consensus.cc:2454] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d659d6399bb04810bcece137244d4bae in term 1.
I20260620 08:16:49.772025 7473 leader_election.cc:302] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce71b96b19cb43169a6eb9df8caadd03, d659d6399bb04810bcece137244d4bae; no voters:
I20260620 08:16:49.772141 7593 raft_consensus.cc:2781] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:16:49.772289 7593 raft_consensus.cc:686] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 LEADER]: Becoming Leader. State: Replica: d659d6399bb04810bcece137244d4bae, State: Running, Role: LEADER
I20260620 08:16:49.772372 7593 consensus_queue.cc:237] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:49.773150 7398 catalog_manager.cc:5495] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae reported cstate change: term changed from 0 to 1, leader changed from <none> to d659d6399bb04810bcece137244d4bae (127.24.249.129). New cstate: current_term: 1 leader_uuid: "d659d6399bb04810bcece137244d4bae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } health_report { overall_health: HEALTHY } } }
I20260620 08:16:49.790335 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.787845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.24.249.129:38265)
0620 08:16:49.787904 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:49.787906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:49.787914 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:16:49.787959 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:49.787959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:49.787976 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:49.788037 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:16:49.788148 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.788688 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.789500 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:49.789725 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:49.789782 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:49.789965 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:16:49.790093 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:16:49.790187 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:49.790211 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 08:16:49.790230 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 08:16:49.791873 7812 raft_consensus.cc:1261] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 1 FOLLOWER]: Refusing update from remote peer d659d6399bb04810bcece137244d4bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:49.791908 7680 raft_consensus.cc:1261] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Refusing update from remote peer d659d6399bb04810bcece137244d4bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:16:49.792232 7875 consensus_queue.cc:1035] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:16:49.792388 7593 consensus_queue.cc:1035] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:16:49.795737 7880 mvcc.cc:205] Tried to move back new op lower bound from 7298840206505062400 to 7298840206428147712. Current Snapshot: MvccSnapshot[applied={T|T < 7298840206505062400}]
I20260620 08:16:49.798373 7399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:16:49.798501 7399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:16:49.799248 7399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Sending DeleteTablet for 3 replicas of tablet 879a4f1bc525407ab56965ff9604723f
I20260620 08:16:49.799551 7522 tablet_service.cc:1514] Processing DeleteTablet for tablet 879a4f1bc525407ab56965ff9604723f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:49 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:33204
I20260620 08:16:49.799633 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet 879a4f1bc525407ab56965ff9604723f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:49 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:33996
I20260620 08:16:49.799710 7792 tablet_service.cc:1514] Processing DeleteTablet for tablet 879a4f1bc525407ab56965ff9604723f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:16:49 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:40484
I20260620 08:16:49.799913 7597 tablet_replica.cc:330] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: stopping tablet replica
I20260620 08:16:49.799979 7597 raft_consensus.cc:2229] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:16:49.799975 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 7437
I20260620 08:16:49.800071 7597 raft_consensus.cc:2258] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:49.800804 7888 tablet_replica.cc:330] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: stopping tablet replica
I20260620 08:16:49.800925 7888 raft_consensus.cc:2229] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:16:49.801026 7888 raft_consensus.cc:2258] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:49.801462 7888 ts_tablet_manager.cc:1905] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:16:49.801540 7889 tablet_replica.cc:330] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: stopping tablet replica
I20260620 08:16:49.801661 7889 raft_consensus.cc:2229] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:16:49.801767 7889 raft_consensus.cc:2258] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:16:49.803218 7888 ts_tablet_manager.cc:1918] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:16:49.803313 7888 log.cc:1196] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/879a4f1bc525407ab56965ff9604723f
I20260620 08:16:49.803586 7888 ts_tablet_manager.cc:1939] T 879a4f1bc525407ab56965ff9604723f P ce71b96b19cb43169a6eb9df8caadd03: Deleting consensus metadata
I20260620 08:16:49.803989 7382 catalog_manager.cc:4841] TS ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899): tablet 879a4f1bc525407ab56965ff9604723f (table test-table [id=49266f35b63f4cc9983de5326965b687]) successfully deleted
W20260620 08:16:49.805804 7383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:38265 (error 108)
I20260620 08:16:49.806003 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 7594
I20260620 08:16:49.806423 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:49.806309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38265 (local address 127.0.0.1:33234)
0620 08:16:49.806356 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:49.806385 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:38265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 08:16:49.806522 7383 catalog_manager.cc:4584] TS d659d6399bb04810bcece137244d4bae (127.24.249.129:38265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 879a4f1bc525407ab56965ff9604723f: Network error: Client connection negotiation failed: client connection to 127.24.249.129:38265: connect: Connection refused (error 111)
I20260620 08:16:49.806289 7889 ts_tablet_manager.cc:1905] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:16:49.808890 7889 ts_tablet_manager.cc:1918] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:16:49.808997 7889 log.cc:1196] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/879a4f1bc525407ab56965ff9604723f
I20260620 08:16:49.809274 7889 ts_tablet_manager.cc:1939] T 879a4f1bc525407ab56965ff9604723f P 21ad019cd89245a5aca2712263d43240: Deleting consensus metadata
I20260620 08:16:49.809737 7385 catalog_manager.cc:4841] TS 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497): tablet 879a4f1bc525407ab56965ff9604723f (table test-table [id=49266f35b63f4cc9983de5326965b687]) successfully deleted
I20260620 08:16:49.811139 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 7729
I20260620 08:16:49.816663 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 7365
I20260620 08:16:49.822602 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34027
--webserver_interface=127.24.249.190
--webserver_port=43159
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:49.906808 7891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:49.907109 7891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:49.907153 7891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:49.908352 7891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:16:49.908432 7891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:49.908457 7891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:49.908480 7891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:16:49.908510 7891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:16:49.910079 7891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=43159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:49.910372 7891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:49.910624 7891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:49.915927 7899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.916067 7896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:49.915922 7897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:49.915961 7891 server_base.cc:1034] running on GCE node
I20260620 08:16:49.916517 7891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:49.916841 7891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:49.918011 7891 hybrid_clock.cc:648] HybridClock initialized: now 1781943409917964 us; error 68 us; skew 500 ppm
Jun 20 08:16:49 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:49.920747 7891 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:16:49.921224 7891 webserver.cc:460] Webserver started at http://127.24.249.190:43159/ using document root <none> and password file <none>
I20260620 08:16:49.921415 7891 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:49.921463 7891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:49.922914 7891 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:49.923753 7906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:49.924045 7891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:49.924160 7891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "5e7734d7d11c4c22b7ff74958fb0a081"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "434c75a05e5858c4f2d1daca81102c95"
server_key_iv: "fbd2ad4355481a6871a6fcdcac474ed5"
server_key_version: "encryptionkey@0"
I20260620 08:16:49.924589 7891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:49.934556 7891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:49.936618 7891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:49.936784 7891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:49.940943 7891 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:34027
I20260620 08:16:49.940976 7958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:34027 every 8 connection(s)
I20260620 08:16:49.941365 7891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:16:49.943215 7959 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:16:49.944142 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Bootstrap starting.
I20260620 08:16:49.948235 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7891
I20260620 08:16:49.948521 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:38265
--local_ip_for_outbound_sockets=127.24.249.129
--tserver_master_addrs=127.24.249.190:34027
--webserver_port=38117
--webserver_interface=127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:16:49.953012 7959 log.cc:824] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:49.955304 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:16:49.955617 7959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081: Bootstrap complete.
I20260620 08:16:49.957912 7959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.958194 7959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7734d7d11c4c22b7ff74958fb0a081, State: Initialized, Role: FOLLOWER
I20260620 08:16:49.958400 7959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.958494 7959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:16:49.958549 7959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:16:49.958604 7959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 2 FOLLOWER]: Advancing to term 3
I20260620 08:16:49.959612 7959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.959735 7959 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7734d7d11c4c22b7ff74958fb0a081; no voters:
I20260620 08:16:49.959939 7959 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 08:16:49.959975 7963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 08:16:49.960090 7963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [term 3 LEADER]: Becoming Leader. State: Replica: 5e7734d7d11c4c22b7ff74958fb0a081, State: Running, Role: LEADER
I20260620 08:16:49.960178 7963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } }
I20260620 08:16:49.960215 7959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:16:49.960434 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7734d7d11c4c22b7ff74958fb0a081. Latest consensus state: current_term: 3 leader_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } } }
I20260620 08:16:49.960510 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:49.960438 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7734d7d11c4c22b7ff74958fb0a081" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34027 } } }
I20260620 08:16:49.960675 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7734d7d11c4c22b7ff74958fb0a081 [sys.catalog]: This master's current role is: LEADER
I20260620 08:16:49.960870 7967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:16:49.961814 7967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:16:49.961951 7967 catalog_manager.cc:1259] Loaded cluster ID: db9798b2eaab4ae5b4fb1635159708f4
I20260620 08:16:49.961990 7967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 08:16:49.962284 7967 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f89477dd9ad google::LogMessage::Fail() at ??:0
@ 0x7f89477e088a google::LogMessage::SendToLog() at ??:0
@ 0x7f89477dd4de google::LogMessage::Flush() at ??:0
@ 0x7f89477dea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f894f6fe0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f894f7078b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f89496ab0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f89496a0954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f894a5046db start_thread at ??:0
@ 0x7f89462db61f clone at ??:0
W20260620 08:16:50.041966 7961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:50.042207 7961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:50.042246 7961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:50.043565 7961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:50.043627 7961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:50.043674 7961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:16:50.045198 7961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:38265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=38117
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:50.045475 7961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:50.045724 7961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:50.051905 7995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:50.051919 7996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:50.052230 7961 server_base.cc:1034] running on GCE node
W20260620 08:16:50.052274 7998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:50.052444 7961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:50.052707 7961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:50.053931 7961 hybrid_clock.cc:648] HybridClock initialized: now 1781943410053902 us; error 43 us; skew 500 ppm
Jun 20 08:16:50 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943410, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:50.056763 7961 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:16:50.057301 7961 webserver.cc:460] Webserver started at http://127.24.249.129:38117/ using document root <none> and password file <none>
I20260620 08:16:50.057560 7961 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:50.057641 7961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:50.059278 7961 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:50.060199 8005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:50.060469 7961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:50.060591 7961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "d659d6399bb04810bcece137244d4bae"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "4e137d31f41d8947b331a8e43bd8186a"
server_key_iv: "1b91694b9a99d781b4a21fa9fbd14676"
server_key_version: "encryptionkey@0"
I20260620 08:16:50.061009 7961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:50.083427 7961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:50.086565 7961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:50.086788 7961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:50.087062 7961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:50.087560 8012 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:16:50.089550 7961 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:16:50.089641 7961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 08:16:50.089692 7961 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:16:50.090416 7961 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:16:50.090515 7961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:16:50.090564 8012 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Bootstrap starting.
I20260620 08:16:50.098032 7961 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:38265
I20260620 08:16:50.098197 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:38265 every 8 connection(s)
I20260620 08:16:50.098487 7961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:16:50.102157 8012 log.cc:824] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Log is configured to *not* fsync() on all Append() calls
I20260620 08:16:50.104224 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 7961
I20260620 08:16:50.104549 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:43899
--local_ip_for_outbound_sockets=127.24.249.130
--tserver_master_addrs=127.24.249.190:34027
--webserver_port=46619
--webserver_interface=127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:16:50.104864 8012 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:16:50.105161 8012 tablet_bootstrap.cc:492] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Bootstrap complete.
I20260620 08:16:50.106401 8012 ts_tablet_manager.cc:1397] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260620 08:16:50.107491 8012 raft_consensus.cc:348] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:50.107753 8012 raft_consensus.cc:729] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d659d6399bb04810bcece137244d4bae, State: Initialized, Role: FOLLOWER
I20260620 08:16:50.107931 8012 consensus_queue.cc:260] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:50.108250 8012 ts_tablet_manager.cc:1428] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260620 08:16:50.218994 8124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:50.219219 8124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:50.219246 8124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:50.220471 8124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:50.220574 8124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:50.220605 8124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:16:50.222210 8124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:43899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=46619
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:50.222468 8124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:50.222729 8124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:50.229300 8130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:50.229298 8133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:50.229585 8131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:50.229841 8124 server_base.cc:1034] running on GCE node
I20260620 08:16:50.230116 8124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:50.230409 8124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260620 08:16:50.230810 8122 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:16:50.099222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:53401)
0620 08:16:50.099401 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.099456 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:50.099964 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:50.230600 (+130636us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
W20260620 08:16:50.231004 8120 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:34027 (0 consecutive failures): Network error: Failed to ping master at 127.24.249.190:34027: Client connection negotiation failed: client connection to 127.24.249.190:34027: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:16:50.231344 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.231200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:32949)
0620 08:16:50.231268 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.231320 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 08:16:50.231570 8124 hybrid_clock.cc:648] HybridClock initialized: now 1781943410231555 us; error 33 us; skew 500 ppm
I20260620 08:16:50.231779 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.231671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:36585)
0620 08:16:50.231710 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.231747 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 08:16:50.231904 8120 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 08:16:50 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943410, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:50.234274 8124 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:16:50.234743 8124 webserver.cc:460] Webserver started at http://127.24.249.130:46619/ using document root <none> and password file <none>
I20260620 08:16:50.234975 8124 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:50.235023 8124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:50.236433 8124 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:16:50.237212 8140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:50.237471 8124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:16:50.237581 8124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "ce71b96b19cb43169a6eb9df8caadd03"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "6fdb98e371512982f920840f5ec00d1f"
server_key_iv: "0009ba0d0bb42be33c0aaba6d8188913"
server_key_version: "encryptionkey@0"
I20260620 08:16:50.238056 8124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:50.255142 8124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:50.257951 8124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:50.258114 8124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:50.258385 8124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:50.258739 8124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:50.258782 8124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:50.258828 8124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:50.258858 8124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:50.265725 8124 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:43899
I20260620 08:16:50.265782 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:43899 every 8 connection(s)
I20260620 08:16:50.266197 8124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:16:50.267544 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.266912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:43483)
0620 08:16:50.267112 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.267197 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
W20260620 08:16:50.267797 8254 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:34027 (0 consecutive failures): Network error: Failed to ping master at 127.24.249.190:34027: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
I20260620 08:16:50.268168 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.268015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:41065)
0620 08:16:50.268115 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.268147 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 08:16:50.268513 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.268447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:43675)
0620 08:16:50.268476 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.268497 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 08:16:50.268669 8254 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:16:50.271726 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 8124
I20260620 08:16:50.272015 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:36497
--local_ip_for_outbound_sockets=127.24.249.131
--tserver_master_addrs=127.24.249.190:34027
--webserver_port=33159
--webserver_interface=127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:16:50.361195 8257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:16:50.361423 8257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:16:50.361449 8257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:16:50.362744 8257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:16:50.362792 8257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:16:50.362821 8257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:16:50.364528 8257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:36497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=33159
--tserver_master_addrs=127.24.249.190:34027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:16:50.364806 8257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:16:50.365061 8257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:16:50.370453 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:50.370491 8257 server_base.cc:1034] running on GCE node
W20260620 08:16:50.370472 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:16:50.370453 8265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:16:50.370857 8257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:16:50.371116 8257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:16:50.372303 8257 hybrid_clock.cc:648] HybridClock initialized: now 1781943410372291 us; error 31 us; skew 500 ppm
Jun 20 08:16:50 dist-test-slave-65w8 krb5kdc[6813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943410, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:16:50.375030 8257 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:16:50.375559 8257 webserver.cc:460] Webserver started at http://127.24.249.131:33159/ using document root <none> and password file <none>
I20260620 08:16:50.375799 8257 fs_manager.cc:359] Metadata directory not provided
I20260620 08:16:50.375869 8257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:16:50.377307 8257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:16:50.378226 8272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:50.378440 8257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:16:50.378558 8257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "21ad019cd89245a5aca2712263d43240"
format_stamp: "Formatted at 2026-06-20 08:16:48 on dist-test-slave-65w8"
server_key: "bf939c6c784120d0bbabbea5b1a9671f"
server_key_iv: "32ae72d43c384ce4a66256324931225a"
server_key_version: "encryptionkey@0"
I20260620 08:16:50.378952 8257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:16:50.408064 8257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:16:50.410482 8257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:16:50.410665 8257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:16:50.410940 8257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:16:50.411271 8257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:16:50.411329 8257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:50.411372 8257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:16:50.411401 8257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:16:50.417745 8257 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:36497
I20260620 08:16:50.417789 8385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:36497 every 8 connection(s)
I20260620 08:16:50.418174 8257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:16:50.419301 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.418855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:54277)
0620 08:16:50.419054 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.419151 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
W20260620 08:16:50.419809 8386 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:34027 (0 consecutive failures): Network error: Failed to ping master at 127.24.249.190:34027: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
I20260620 08:16:50.420212 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.420066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:36715)
0620 08:16:50.420161 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.420190 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 08:16:50.420683 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:50.420557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:42269)
0620 08:16:50.420631 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:50.420661 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 08:16:50.420881 8386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:16:50.428335 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 8257
I20260620 08:16:51.232729 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:51.232406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:38979)
0620 08:16:51.232635 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:51.232679 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260620 08:16:51.269289 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:51.269049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:59601)
0620 08:16:51.269191 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:51.269231 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260620 08:16:51.295702 8392 raft_consensus.cc:482] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:51.295825 8392 raft_consensus.cc:504] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:51.296159 8392 leader_election.cc:288] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497), ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
Jun 20 08:16:51 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943410, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.130@KRBTEST.COM
Jun 20 08:16:51 dist-test-slave-65w8 krb5kdc[6813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943410, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.131@KRBTEST.COM
I20260620 08:16:51.307415 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:51.296416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:43899 (local address 127.24.249.129:39139)
0620 08:16:51.296559 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:51.296571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:51.296615 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:51.297620 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:51.297621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:51.297913 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:51.298131 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:51.298137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.299360 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.299360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:51.300097 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:51.300100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.300746 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.300747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:51.300864 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:51.301485 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:51.301492 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:51.303327 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:51.305579 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:51.305586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:51.305589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:51.306029 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:51.306323 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:51.306323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:51.306324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:51.306447 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:51.306892 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:51.306893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:51.307026 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 08:16:51.307093 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:16:51.307308 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":55,"thread_start_us":79,"threads_started":1}
I20260620 08:16:51.307447 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:51.296443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:47999 (local address 127.24.249.131:36497)
0620 08:16:51.296679 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:51.296686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:51.296709 (+ 23us) server_negotiation.cc:407] Connection header received
0620 08:16:51.297272 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:51.297275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:51.297339 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:51.297505 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:51.298285 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.299156 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.300204 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.300639 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.300771 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:51.303884 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:51.303888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:51.303888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:51.303898 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:51.305670 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:51.306159 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:51.306161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:51.306162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:51.306245 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:51.306558 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:51.306559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:51.306559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:51.306787 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:51.306870 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:51.307220 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 08:16:51.307347 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
I20260620 08:16:51.307507 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:51.296501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:39139 (local address 127.24.249.130:43899)
0620 08:16:51.296669 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 08:16:51.296682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 08:16:51.296713 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:16:51.297287 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:16:51.297290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:16:51.297339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:16:51.297494 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:16:51.298257 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.299239 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.300213 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.300645 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.300776 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:51.303463 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:16:51.303468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:16:51.303468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:16:51.303481 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:16:51.305486 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:51.306115 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:51.306119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:51.306119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:51.306244 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:16:51.306530 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:16:51.306531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:16:51.306531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:16:51.306792 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:16:51.306895 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 08:16:51.307200 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 08:16:51.307377 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20260620 08:16:51.307539 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:51.296332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:36497 (local address 127.24.249.129:47999)
0620 08:16:51.296392 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:51.296408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:16:51.296473 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:16:51.297556 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:16:51.297559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:16:51.297949 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:16:51.298133 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:51.298139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.299273 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.299275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:51.300099 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:16:51.300101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:16:51.300746 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:16:51.300748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:16:51.300866 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:16:51.301556 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:16:51.301561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:16:51.303766 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:16:51.305747 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:51.305748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:51.305748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:51.306052 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:51.306325 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:16:51.306326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:16:51.306326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:16:51.306452 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:16:51.306863 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:16:51.306870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:16:51.307010 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 08:16:51.307131 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 08:16:51.307397 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":169}
W20260620 08:16:51.308918 8006 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
W20260620 08:16:51.309003 8009 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
I20260620 08:16:51.309078 8009 leader_election.cc:302] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d659d6399bb04810bcece137244d4bae; no voters: 21ad019cd89245a5aca2712263d43240, ce71b96b19cb43169a6eb9df8caadd03
I20260620 08:16:51.309206 8392 raft_consensus.cc:2726] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:16:51.421526 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:51.421267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:56481)
0620 08:16:51.421433 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:51.421484 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260620 08:16:52.233757 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:52.233469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:54039)
0620 08:16:52.233662 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:52.233710 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260620 08:16:52.270143 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:52.269851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:43927)
0620 08:16:52.270041 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:52.270099 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260620 08:16:52.422386 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:52.422162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:60255)
0620 08:16:52.422290 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:52.422329 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260620 08:16:53.234684 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:53.234426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:51681)
0620 08:16:53.234589 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:53.234634 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260620 08:16:53.270890 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:53.270663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:54997)
0620 08:16:53.270804 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:53.270847 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260620 08:16:53.312597 8402 raft_consensus.cc:482] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:53.312705 8402 raft_consensus.cc:504] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:53.312896 8402 leader_election.cc:288] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497), ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
W20260620 08:16:53.313364 8009 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
W20260620 08:16:53.313432 8006 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
I20260620 08:16:53.313520 8006 leader_election.cc:302] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d659d6399bb04810bcece137244d4bae; no voters: 21ad019cd89245a5aca2712263d43240, ce71b96b19cb43169a6eb9df8caadd03
I20260620 08:16:53.313653 8402 raft_consensus.cc:2726] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:16:53.423293 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:53.423048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:33027)
0620 08:16:53.423203 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:53.423244 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260620 08:16:54.235702 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:54.235403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:36795)
0620 08:16:54.235599 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:54.235646 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":63,"threads_started":1}
I20260620 08:16:54.271795 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:54.271509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:36409)
0620 08:16:54.271696 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:54.271739 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":138,"threads_started":1}
I20260620 08:16:54.424100 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:54.423879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:58253)
0620 08:16:54.424014 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:54.424055 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260620 08:16:55.236618 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:55.236374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:49071)
0620 08:16:55.236527 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:55.236571 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260620 08:16:55.272521 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:55.272293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:59273)
0620 08:16:55.272433 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:55.272478 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260620 08:16:55.385252 8409 raft_consensus.cc:482] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:55.385388 8409 raft_consensus.cc:504] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:55.385653 8409 leader_election.cc:288] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497), ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
W20260620 08:16:55.386097 8009 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
W20260620 08:16:55.386193 8006 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
I20260620 08:16:55.386257 8006 leader_election.cc:302] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d659d6399bb04810bcece137244d4bae; no voters: 21ad019cd89245a5aca2712263d43240, ce71b96b19cb43169a6eb9df8caadd03
I20260620 08:16:55.386334 8409 raft_consensus.cc:2726] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:16:55.424907 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:55.424689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:37795)
0620 08:16:55.424819 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:55.424860 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260620 08:16:56.237468 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:56.237209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:45999)
0620 08:16:56.237368 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:56.237415 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260620 08:16:56.273371 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:56.273159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:55923)
0620 08:16:56.273292 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:56.273328 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260620 08:16:56.425699 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:56.425459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:35617)
0620 08:16:56.425607 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:56.425653 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260620 08:16:57.238447 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:57.238208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:60915)
0620 08:16:57.238349 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:57.238399 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260620 08:16:57.274118 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:57.273856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:51921)
0620 08:16:57.274040 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:57.274072 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260620 08:16:57.426644 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:57.426353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:34115)
0620 08:16:57.426509 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:57.426596 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260620 08:16:57.498301 8417 raft_consensus.cc:482] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:16:57.498418 8417 raft_consensus.cc:504] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:16:57.498684 8417 leader_election.cc:288] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497), ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
W20260620 08:16:57.499135 8006 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
W20260620 08:16:57.499323 8009 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
I20260620 08:16:57.499379 8009 leader_election.cc:302] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d659d6399bb04810bcece137244d4bae; no voters: 21ad019cd89245a5aca2712263d43240, ce71b96b19cb43169a6eb9df8caadd03
I20260620 08:16:57.499468 8417 raft_consensus.cc:2726] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:16:58.239290 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:58.239061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:50825)
0620 08:16:58.239208 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:58.239243 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260620 08:16:58.274952 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:58.274720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:53521)
0620 08:16:58.274860 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:58.274900 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260620 08:16:58.427470 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:58.427220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:44667)
0620 08:16:58.427379 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:58.427417 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20260620 08:16:59.240235 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:59.239933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:47017)
0620 08:16:59.240142 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:59.240187 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
I20260620 08:16:59.275792 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:59.275528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:55639)
0620 08:16:59.275706 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:59.275744 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260620 08:16:59.428248 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:16:59.427974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:43063)
0620 08:16:59.428155 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:16:59.428200 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260620 08:17:00.241173 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:00.240901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:34457)
0620 08:17:00.241064 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:00.241108 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260620 08:17:00.276639 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:00.276337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:40473)
0620 08:17:00.276545 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:00.276588 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260620 08:17:00.429255 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:00.428982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:52407)
0620 08:17:00.429155 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:00.429204 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260620 08:17:01.241977 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:01.241690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:54775)
0620 08:17:01.241816 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:01.241854 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260620 08:17:01.277390 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:01.277193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:39295)
0620 08:17:01.277309 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:01.277345 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260620 08:17:01.430243 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:01.429998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:45855)
0620 08:17:01.430150 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:01.430195 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260620 08:17:02.242697 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:02.242429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:55653)
0620 08:17:02.242603 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:02.242650 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260620 08:17:02.278264 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:02.277978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:39051)
0620 08:17:02.278168 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:02.278214 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260620 08:17:02.431290 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:02.430969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:51023)
0620 08:17:02.431166 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:02.431219 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260620 08:17:03.243496 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:03.243242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:40485)
0620 08:17:03.243382 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:03.243449 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260620 08:17:03.279037 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:03.278768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:55861)
0620 08:17:03.278958 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:03.278991 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260620 08:17:03.432126 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:03.431862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:57861)
0620 08:17:03.432025 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:03.432071 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260620 08:17:04.244343 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:04.244100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:46821)
0620 08:17:04.244249 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:04.244296 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260620 08:17:04.279776 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:04.279571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:35083)
0620 08:17:04.279693 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:04.279730 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260620 08:17:04.316757 8438 raft_consensus.cc:482] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:17:04.316877 8438 raft_consensus.cc:504] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad019cd89245a5aca2712263d43240" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 36497 } } peers { permanent_uuid: "ce71b96b19cb43169a6eb9df8caadd03" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 43899 } } peers { permanent_uuid: "d659d6399bb04810bcece137244d4bae" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38265 } }
I20260620 08:17:04.317123 8438 leader_election.cc:288] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497), ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899)
W20260620 08:17:04.317531 8009 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21ad019cd89245a5aca2712263d43240 (127.24.249.131:36497): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
W20260620 08:17:04.317646 8006 leader_election.cc:341] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce71b96b19cb43169a6eb9df8caadd03 (127.24.249.130:43899): Not found: Tablet not found: 879a4f1bc525407ab56965ff9604723f
I20260620 08:17:04.317727 8006 leader_election.cc:302] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d659d6399bb04810bcece137244d4bae; no voters: 21ad019cd89245a5aca2712263d43240, ce71b96b19cb43169a6eb9df8caadd03
I20260620 08:17:04.317838 8438 raft_consensus.cc:2726] T 879a4f1bc525407ab56965ff9604723f P d659d6399bb04810bcece137244d4bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:17:04.432973 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:04.432687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:48823)
0620 08:17:04.432887 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:04.432926 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":146,"threads_started":1}
I20260620 08:17:05.245334 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:05.245059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.129:48819)
0620 08:17:05.245242 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:05.245279 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260620 08:17:05.280643 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:05.280365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.130:56689)
0620 08:17:05.280548 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:05.280593 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260620 08:17:05.433935 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:05.433616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34027 (local address 127.24.249.131:54275)
0620 08:17:05.433767 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:05.433816 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:34027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260620 08:17:05.436182 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 7961
I20260620 08:17:05.441713 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 8124
I20260620 08:17:05.446236 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 8257
2026-06-20T08:17:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18513 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:17:05 dist-test-slave-65w8 krb5kdc[8445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:17:05 dist-test-slave-65w8 krb5kdc[8445](info): set up 2 sockets
Jun 20 08:17:05 dist-test-slave-65w8 krb5kdc[8445](info): commencing operation
krb5kdc: starting...
W20260620 08:17:06.554301 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:17:06 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:17:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:17:06.682487 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41031
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:17:06.766852 8458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:06.767115 8458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:06.767156 8458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:06.768302 8458 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:17:06.768369 8458 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:17:06.768391 8458 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:17:06.768410 8458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:17:06.768443 8458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:06.768544 8458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:06.768592 8458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:17:06.768616 8458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:17:06.770183 8458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:06.770478 8458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:06.770720 8458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:06.776533 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:06.776454 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:06.776463 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:06.776479 8458 server_base.cc:1034] running on GCE node
I20260620 08:17:06.777212 8458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:06.777637 8458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:06.778932 8458 hybrid_clock.cc:648] HybridClock initialized: now 1781943426778904 us; error 41 us; skew 500 ppm
Jun 20 08:17:06 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943426, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:06.781840 8458 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:17:06.782454 8458 webserver.cc:460] Webserver started at http://127.24.249.190:44185/ using document root <none> and password file <none>
I20260620 08:17:06.782727 8458 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:06.782796 8458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:06.782948 8458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:17:06.784086 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "7411ba9266644f079b91282af73f2ed0"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "3001c0ee719ea712f80ef9fe2a2ca792"
server_key_iv: "0e5da18418f78dcd8f030c7f42f8f86f"
server_key_version: "encryptionkey@0"
I20260620 08:17:06.784451 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "7411ba9266644f079b91282af73f2ed0"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "3001c0ee719ea712f80ef9fe2a2ca792"
server_key_iv: "0e5da18418f78dcd8f030c7f42f8f86f"
server_key_version: "encryptionkey@0"
I20260620 08:17:06.785940 8458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:17:06.786791 8473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:06.787045 8458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:17:06.787159 8458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "7411ba9266644f079b91282af73f2ed0"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "3001c0ee719ea712f80ef9fe2a2ca792"
server_key_iv: "0e5da18418f78dcd8f030c7f42f8f86f"
server_key_version: "encryptionkey@0"
I20260620 08:17:06.787247 8458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:06.799381 8458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:06.801545 8458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:06.801724 8458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:06.805809 8458 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:41031
I20260620 08:17:06.805920 8525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:41031 every 8 connection(s)
I20260620 08:17:06.806255 8458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:17:06.807101 8526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:17:06.807897 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 8458
I20260620 08:17:06.808003 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:17:06.808198 25574 external_mini_cluster.cc:1428] Setting key 1a2beac45bb48d38d224d3d400068db8
I20260620 08:17:06.810561 8526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Bootstrap starting.
I20260620 08:17:06.811211 8526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Neither blocks nor log segments found. Creating new log.
I20260620 08:17:06.811507 8526 log.cc:824] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:06.812263 8526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: No bootstrap required, opened a new log
I20260620 08:17:06.814766 8526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:06.814944 8526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:17:06.815011 8526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7411ba9266644f079b91282af73f2ed0, State: Initialized, Role: FOLLOWER
I20260620 08:17:06.815161 8526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:06.815253 8526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:17:06.815296 8526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:17:06.815351 8526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 08:17:06 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:17:06.816183 8526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:06.816319 8526 leader_election.cc:302] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7411ba9266644f079b91282af73f2ed0; no voters:
I20260620 08:17:06.816535 8526 leader_election.cc:288] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:17:06.816582 8531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:17:06.816766 8531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 LEADER]: Becoming Leader. State: Replica: 7411ba9266644f079b91282af73f2ed0, State: Running, Role: LEADER
I20260620 08:17:06.816859 8526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:17:06.816912 8531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:06.817621 8533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7411ba9266644f079b91282af73f2ed0. Latest consensus state: current_term: 1 leader_uuid: "7411ba9266644f079b91282af73f2ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } } }
I20260620 08:17:06.817699 8533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:17:06.817945 8532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7411ba9266644f079b91282af73f2ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } } }
I20260620 08:17:06.818073 8532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:17:06.818225 8540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:17:06.818758 8540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:17:06.820160 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:06.808816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.24.249.190:41031)
0620 08:17:06.809028 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:06.809044 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:06.809070 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:17:06.809513 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:06.809519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:06.809567 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:06.809680 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:06.810574 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:06.811727 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:06.812589 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:06.813253 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:06.813380 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:06.815988 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:06.815992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:06.815992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:06.816002 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:06.818265 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:06.818747 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:06.818750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:06.818750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:06.818916 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:06.819197 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:06.819198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:06.819198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:06.819500 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:06.819709 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:06.819778 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 08:17:06.819917 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":127,"threads_started":1}
I20260620 08:17:06.820770 8540 catalog_manager.cc:1347] Generated new cluster ID: 5d50758508c140c7b43ca3d33d4e11ed
I20260620 08:17:06.820847 8540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:17:06.834884 8540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:17:06.835624 8540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:17:06.838898 8540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Generated new TSK 0
I20260620 08:17:06.839080 8540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:17:06.872984 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41031
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:17:06.956475 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:06.956722 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:06.956763 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:06.957991 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:06.958046 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:06.958087 8552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:17:06.959555 8552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:06.959816 8552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:06.960048 8552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:06.965265 8558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:06.965281 8557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:06.965299 8560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:06.965423 8552 server_base.cc:1034] running on GCE node
I20260620 08:17:06.965766 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:06.966063 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:06.967206 8552 hybrid_clock.cc:648] HybridClock initialized: now 1781943426967195 us; error 33 us; skew 500 ppm
Jun 20 08:17:06 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943426, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:06.969798 8552 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:17:06.970340 8552 webserver.cc:460] Webserver started at http://127.24.249.129:44103/ using document root <none> and password file <none>
I20260620 08:17:06.970573 8552 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:06.970640 8552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:06.970777 8552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:17:06.971828 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "92fa5034c38043c3a4621b3c60f5962e"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "891fa6637a8fb14e91714be88cffbc6c"
server_key_iv: "97f106ab6f8d4bdbd3adacc207433ebe"
server_key_version: "encryptionkey@0"
I20260620 08:17:06.972249 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "92fa5034c38043c3a4621b3c60f5962e"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "891fa6637a8fb14e91714be88cffbc6c"
server_key_iv: "97f106ab6f8d4bdbd3adacc207433ebe"
server_key_version: "encryptionkey@0"
I20260620 08:17:06.973731 8552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:17:06.974623 8567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:06.974834 8552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:06.974936 8552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "92fa5034c38043c3a4621b3c60f5962e"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "891fa6637a8fb14e91714be88cffbc6c"
server_key_iv: "97f106ab6f8d4bdbd3adacc207433ebe"
server_key_version: "encryptionkey@0"
I20260620 08:17:06.975021 8552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:07.000555 8552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:07.003191 8552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:07.003396 8552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:07.003700 8552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:07.004041 8552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:17:07.004096 8552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.004137 8552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:17:07.004165 8552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.010624 8552 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:45343
I20260620 08:17:07.010671 8680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:45343 every 8 connection(s)
I20260620 08:17:07.011023 8552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
Jun 20 08:17:07 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943426, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:17:07.018918 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 8552
I20260620 08:17:07.019070 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:17:07.019284 25574 external_mini_cluster.cc:1428] Setting key a3358c4950a59b64bb5b61c2a6d59646
I20260620 08:17:07.021760 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.011840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:54935 (local address 127.24.249.190:41031)
0620 08:17:07.011934 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.011937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.012271 (+ 334us) server_negotiation.cc:407] Connection header received
0620 08:17:07.012830 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.012831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.012842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.012903 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:07.013986 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.014540 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.015590 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.015800 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.015867 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.018446 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:07.018448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:07.018448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:07.018456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:07.020295 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.020718 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.020719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.020719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.020783 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.021070 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.021071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.021072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.021195 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:07.021269 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.021571 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.021665 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:17:07.021965 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.011829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:54935)
0620 08:17:07.012118 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.012172 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.012712 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.013007 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.013011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.013640 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:07.013854 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.013877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.014655 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.014656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.015472 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.015474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.015891 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.015892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.016009 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.016648 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:07.016656 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:07.018342 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:07.020393 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:07.020401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:07.020404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:07.020639 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:07.020852 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:07.020853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:07.020854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:07.020975 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:07.021272 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:07.021280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:07.021427 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.021536 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.021702 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":138,"threads_started":1}
I20260620 08:17:07.022480 8681 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41031
I20260620 08:17:07.022617 8681 heartbeater.cc:464] Registering TS with master...
I20260620 08:17:07.022820 8681 heartbeater.cc:511] Master 127.24.249.190:41031 requested a full tablet report, sending...
I20260620 08:17:07.023343 8490 ts_manager.cc:194] Registered new tserver with Master: 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129:45343)
I20260620 08:17:07.024009 8490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:54935
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:17:07.049319 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41031
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:17:07.133419 8686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:07.133633 8686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:07.133657 8686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:07.134984 8686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:07.135025 8686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:07.135052 8686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:17:07.136802 8686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:07.137012 8686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:07.137183 8686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:07.142180 8694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:07.142166 8692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:07.142395 8686 server_base.cc:1034] running on GCE node
W20260620 08:17:07.142270 8691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:07.142717 8686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:07.142983 8686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:07.144150 8686 hybrid_clock.cc:648] HybridClock initialized: now 1781943427144127 us; error 39 us; skew 500 ppm
Jun 20 08:17:07 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:07.146787 8686 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:17:07.147215 8686 webserver.cc:460] Webserver started at http://127.24.249.130:42121/ using document root <none> and password file <none>
I20260620 08:17:07.147446 8686 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:07.147521 8686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:07.147675 8686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:17:07.148733 8686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "6bb308b20835493183492d5d75f8d094"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "11e5b52d35c7557a72ede20118df0746"
server_key_iv: "dfa1d701bde3613cf17f05b57b0d9df4"
server_key_version: "encryptionkey@0"
I20260620 08:17:07.149099 8686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "6bb308b20835493183492d5d75f8d094"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "11e5b52d35c7557a72ede20118df0746"
server_key_iv: "dfa1d701bde3613cf17f05b57b0d9df4"
server_key_version: "encryptionkey@0"
I20260620 08:17:07.150573 8686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:17:07.151351 8701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.151532 8686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:07.151636 8686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "6bb308b20835493183492d5d75f8d094"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "11e5b52d35c7557a72ede20118df0746"
server_key_iv: "dfa1d701bde3613cf17f05b57b0d9df4"
server_key_version: "encryptionkey@0"
I20260620 08:17:07.151710 8686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:07.164130 8686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:07.166282 8686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:07.166473 8686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:07.166760 8686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:07.167095 8686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:17:07.167152 8686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.167196 8686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:17:07.167222 8686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.173257 8686 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:45787
I20260620 08:17:07.173324 8814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:45787 every 8 connection(s)
I20260620 08:17:07.173672 8686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:17:07.174896 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 8686
I20260620 08:17:07.174997 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:17:07.175185 25574 external_mini_cluster.cc:1428] Setting key 3bcf9f071fed7f5058c7c82b32f52d6c
Jun 20 08:17:07 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
I20260620 08:17:07.185108 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.174443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:41645 (local address 127.24.249.190:41031)
0620 08:17:07.174534 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.174580 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.174945 (+ 365us) server_negotiation.cc:407] Connection header received
0620 08:17:07.175688 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.175690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.175702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.175769 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:07.177139 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.177848 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.178921 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.179247 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.179325 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.181843 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:07.181845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:07.181846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:07.181854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:07.183622 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.184115 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.184116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.184117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.184177 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.184428 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.184428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.184429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.184565 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:07.184635 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.184943 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.185028 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:17:07.185299 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.174417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:41645)
0620 08:17:07.174781 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.174833 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.175574 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.175821 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.175845 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.176686 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:07.177011 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.177018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.177970 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.177971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.178775 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.178777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.179340 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.179341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.179444 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.180013 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:07.180021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:07.181743 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:07.183722 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:07.183729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:07.183731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:07.184018 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:07.184253 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:07.184254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:07.184254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:07.184357 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:07.184633 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:07.184638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:07.184806 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.184906 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.185056 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":113,"threads_started":1}
I20260620 08:17:07.186138 8815 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41031
I20260620 08:17:07.186266 8815 heartbeater.cc:464] Registering TS with master...
I20260620 08:17:07.186466 8815 heartbeater.cc:511] Master 127.24.249.190:41031 requested a full tablet report, sending...
I20260620 08:17:07.186902 8490 ts_manager.cc:194] Registered new tserver with Master: 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787)
I20260620 08:17:07.187301 8490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:41645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:17:07.202828 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41031
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:17:07.287524 8820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:07.287770 8820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:07.287812 8820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:07.289022 8820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:07.289098 8820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:07.289146 8820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:17:07.290705 8820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:07.290980 8820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:07.291220 8820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:07.296516 8826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:07.296525 8828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:07.296627 8820 server_base.cc:1034] running on GCE node
W20260620 08:17:07.296528 8825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:07.296965 8820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:07.297263 8820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:07.298480 8820 hybrid_clock.cc:648] HybridClock initialized: now 1781943427298498 us; error 77 us; skew 500 ppm
Jun 20 08:17:07 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:07.301262 8820 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:17:07.301779 8820 webserver.cc:460] Webserver started at http://127.24.249.131:33745/ using document root <none> and password file <none>
I20260620 08:17:07.302048 8820 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:07.302119 8820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:07.302266 8820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:17:07.303386 8820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "06336fae806e4996afa608f118e4d260"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "74b69a91421add1cad0021ab823d716a"
server_key_iv: "5311e5df2896c7c650f40ea1f0f2b03e"
server_key_version: "encryptionkey@0"
I20260620 08:17:07.303754 8820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "06336fae806e4996afa608f118e4d260"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "74b69a91421add1cad0021ab823d716a"
server_key_iv: "5311e5df2896c7c650f40ea1f0f2b03e"
server_key_version: "encryptionkey@0"
I20260620 08:17:07.305130 8820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:17:07.306010 8835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.306231 8820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:07.306360 8820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "06336fae806e4996afa608f118e4d260"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "74b69a91421add1cad0021ab823d716a"
server_key_iv: "5311e5df2896c7c650f40ea1f0f2b03e"
server_key_version: "encryptionkey@0"
I20260620 08:17:07.306448 8820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:07.326730 8820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:07.329294 8820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:07.329514 8820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:07.329803 8820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:07.330194 8820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:17:07.330258 8820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.330309 8820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:17:07.330343 8820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:07.336796 8820 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:35119
I20260620 08:17:07.336877 8948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:35119 every 8 connection(s)
I20260620 08:17:07.337216 8820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:17:07.338385 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 8820
I20260620 08:17:07.338485 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:17:07.338661 25574 external_mini_cluster.cc:1428] Setting key 5e9cb0bb6830f736872a0b81a8175b40
Jun 20 08:17:07 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:17:07.348065 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.338231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:60953 (local address 127.24.249.190:41031)
0620 08:17:07.338303 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.338305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.339118 (+ 813us) server_negotiation.cc:407] Connection header received
0620 08:17:07.339719 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.339720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.339731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.339772 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:07.340771 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.341310 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.342520 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.342719 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.342779 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.345164 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:07.345165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:07.345166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:07.345173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:07.346635 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.347070 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.347071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.347071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.347123 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.347405 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.347406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.347406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.347542 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:07.347621 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.347909 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.347996 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:17:07.348234 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.338118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:60953)
0620 08:17:07.338969 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.339021 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.339636 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.339836 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.339840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.340465 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:07.340673 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.340678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.341415 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.341416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.342401 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.342403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.342824 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.342825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.342946 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.343500 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:07.343508 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:07.345033 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:07.346731 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:07.346738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:07.346740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:07.346990 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:07.347202 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:07.347204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:07.347204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:07.347321 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:07.347641 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:07.347648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:07.347777 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.347877 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.348014 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":103,"threads_started":1}
I20260620 08:17:07.348834 8949 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41031
I20260620 08:17:07.348924 8949 heartbeater.cc:464] Registering TS with master...
I20260620 08:17:07.349071 8949 heartbeater.cc:511] Master 127.24.249.190:41031 requested a full tablet report, sending...
I20260620 08:17:07.349447 8490 ts_manager.cc:194] Registered new tserver with Master: 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
I20260620 08:17:07.350123 8490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:60953
I20260620 08:17:07.350950 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:17:07.359462 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.352545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.24.249.190:41031)
0620 08:17:07.352594 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.352596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.352651 (+ 55us) server_negotiation.cc:407] Connection header received
0620 08:17:07.352701 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.352701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.352710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.352765 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:07.353332 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.353951 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.354747 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.354932 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.354988 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.356290 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:07.356291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:07.356294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:07.356301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:07.357697 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.358380 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.358381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.358382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.358437 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:07.358795 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:07.358795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:07.358796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:07.358961 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:07.359073 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.359287 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.359365 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:17:07.361485 8490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:17:07.363277 8490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:17:07.370735 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.366365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.24.249.131:35119)
0620 08:17:07.366555 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.366559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.367058 (+ 499us) server_negotiation.cc:407] Connection header received
0620 08:17:07.367142 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.367144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.367160 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.367244 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:07.367365 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.368293 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.369477 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.370251 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.370338 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.370399 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.370602 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.370644 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":121,"threads_started":1}
I20260620 08:17:07.370744 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.365935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45787 (local address 127.0.0.1:41948)
0620 08:17:07.366131 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.366173 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.366238 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.366573 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.366575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.366577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:07.366649 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.366651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.368020 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.368021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.369482 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.369485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.370372 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.370372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.370469 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.370484 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.370627 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.370667 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":79,"thread_start_us":71,"threads_started":1}
I20260620 08:17:07.370744 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.366160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35119 (local address 127.0.0.1:39012)
0620 08:17:07.366992 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.367003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.367056 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.367262 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.367263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.367265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:07.367307 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.367309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.368390 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.368392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.369389 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.369391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.370369 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.370371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.370469 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.370484 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.370576 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.370649 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":22,"thread_start_us":70,"threads_started":1}
I20260620 08:17:07.371244 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.366185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.24.249.130:45787)
0620 08:17:07.366381 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.366386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.366396 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:17:07.366452 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.366454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.366466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.366553 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:07.366735 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.367891 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.369556 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.370270 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.370338 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.370403 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.370968 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.371013 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260620 08:17:07.371706 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.365935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:45343 (local address 127.0.0.1:48596)
0620 08:17:07.366253 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.366265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.366324 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.366972 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.366973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.366974 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:07.367021 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.367023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.368040 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.368041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.369482 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.369485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.371448 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.371450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.371526 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.371530 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.371585 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.371630 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":112,"thread_start_us":71,"threads_started":1}
I20260620 08:17:07.371721 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.366075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.24.249.129:45343)
0620 08:17:07.366727 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.366731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.366740 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:17:07.366780 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.366782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.366798 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.366887 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:07.367102 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.367910 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.370440 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.371271 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.371339 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.371411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.371600 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.371641 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":87,"threads_started":1}
I20260620 08:17:07.371783 8749 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac1220c67934cb281abea359e7873b3 (DEFAULT_TABLE table=test-table [id=a07db92d5b994d64944c7512d5a11724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:17:07.371784 8883 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac1220c67934cb281abea359e7873b3 (DEFAULT_TABLE table=test-table [id=a07db92d5b994d64944c7512d5a11724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:17:07.372148 8749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac1220c67934cb281abea359e7873b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:17:07.372148 8883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac1220c67934cb281abea359e7873b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:17:07.372665 8615 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac1220c67934cb281abea359e7873b3 (DEFAULT_TABLE table=test-table [id=a07db92d5b994d64944c7512d5a11724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:17:07.372990 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac1220c67934cb281abea359e7873b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:17:07.375751 8967 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Bootstrap starting.
I20260620 08:17:07.375756 8966 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap starting.
I20260620 08:17:07.375885 8968 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Bootstrap starting.
I20260620 08:17:07.376466 8966 tablet_bootstrap.cc:654] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Neither blocks nor log segments found. Creating new log.
I20260620 08:17:07.376492 8968 tablet_bootstrap.cc:654] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Neither blocks nor log segments found. Creating new log.
I20260620 08:17:07.376492 8967 tablet_bootstrap.cc:654] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Neither blocks nor log segments found. Creating new log.
I20260620 08:17:07.376996 8966 log.cc:824] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:07.377102 8968 log.cc:824] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:07.377102 8967 log.cc:824] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:07.377883 8968 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: No bootstrap required, opened a new log
I20260620 08:17:07.377902 8966 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: No bootstrap required, opened a new log
I20260620 08:17:07.377846 8967 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: No bootstrap required, opened a new log
I20260620 08:17:07.377959 8968 ts_tablet_manager.cc:1397] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:17:07.377959 8966 ts_tablet_manager.cc:1397] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:17:07.377976 8967 ts_tablet_manager.cc:1397] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:17:07.379465 8967 raft_consensus.cc:348] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.379601 8967 raft_consensus.cc:374] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:17:07.379621 8967 raft_consensus.cc:729] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06336fae806e4996afa608f118e4d260, State: Initialized, Role: FOLLOWER
I20260620 08:17:07.379748 8967 consensus_queue.cc:260] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.380023 8949 heartbeater.cc:503] Master 127.24.249.190:41031 was elected leader, sending a full tablet report...
I20260620 08:17:07.380033 8967 ts_tablet_manager.cc:1428] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:17:07.380003 8966 raft_consensus.cc:348] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.380003 8968 raft_consensus.cc:348] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.380185 8968 raft_consensus.cc:374] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:17:07.380184 8966 raft_consensus.cc:374] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:17:07.380221 8968 raft_consensus.cc:729] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb308b20835493183492d5d75f8d094, State: Initialized, Role: FOLLOWER
I20260620 08:17:07.380221 8966 raft_consensus.cc:729] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fa5034c38043c3a4621b3c60f5962e, State: Initialized, Role: FOLLOWER
I20260620 08:17:07.380337 8968 consensus_queue.cc:260] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.380348 8966 consensus_queue.cc:260] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.380753 8966 ts_tablet_manager.cc:1428] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:17:07.380748 8968 ts_tablet_manager.cc:1428] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:17:07.380766 8815 heartbeater.cc:503] Master 127.24.249.190:41031 was elected leader, sending a full tablet report...
I20260620 08:17:07.380892 8681 heartbeater.cc:503] Master 127.24.249.190:41031 was elected leader, sending a full tablet report...
W20260620 08:17:07.424325 8816 tablet.cc:2369] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:17:07.511850 8682 tablet.cc:2369] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:17:07.587868 8950 tablet.cc:2369] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:17:07.599263 8974 raft_consensus.cc:482] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:17:07.599432 8974 raft_consensus.cc:504] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.599829 8974 leader_election.cc:288] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
I20260620 08:17:07.604274 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.600187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:47857 (local address 127.24.249.130:45787)
0620 08:17:07.600263 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.600266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.600348 (+ 82us) server_negotiation.cc:407] Connection header received
0620 08:17:07.600453 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.600454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.600466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.600507 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:07.600814 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.601581 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.603191 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.603819 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.603908 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.603917 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.604159 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.604202 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:17:07.604274 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.600455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:58941 (local address 127.24.249.131:35119)
0620 08:17:07.600512 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.600515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.600524 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:17:07.600656 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.600657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.600664 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.600844 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:07.601007 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.601691 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.603191 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.603832 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.603908 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.603917 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.604159 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.604202 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 08:17:07.604326 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.600062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45787 (local address 127.24.249.129:47857)
0620 08:17:07.600259 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.600273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.600365 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.600541 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.600542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.600544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:07.600631 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.600633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.601694 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.601695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.603035 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.603038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.603949 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.603950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.604058 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.604060 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.604136 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.604195 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":24,"thread_start_us":120,"threads_started":1}
I20260620 08:17:07.604326 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.600315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35119 (local address 127.24.249.129:58941)
0620 08:17:07.600472 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.600482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:07.600538 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:07.600862 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:07.600863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:07.600865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:07.600932 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.600934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.601789 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.601789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.603034 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:07.603038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.603951 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.603952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:07.604054 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.604060 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 08:17:07.604136 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 08:17:07.604195 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":15,"thread_start_us":83,"threads_started":1}
I20260620 08:17:07.604771 8769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac1220c67934cb281abea359e7873b3" candidate_uuid: "92fa5034c38043c3a4621b3c60f5962e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb308b20835493183492d5d75f8d094" is_pre_election: true
I20260620 08:17:07.604964 8769 raft_consensus.cc:2454] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92fa5034c38043c3a4621b3c60f5962e in term 0.
I20260620 08:17:07.604976 8903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac1220c67934cb281abea359e7873b3" candidate_uuid: "92fa5034c38043c3a4621b3c60f5962e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06336fae806e4996afa608f118e4d260" is_pre_election: true
I20260620 08:17:07.605139 8903 raft_consensus.cc:2454] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92fa5034c38043c3a4621b3c60f5962e in term 0.
I20260620 08:17:07.605415 8568 leader_election.cc:302] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06336fae806e4996afa608f118e4d260, 92fa5034c38043c3a4621b3c60f5962e; no voters:
I20260620 08:17:07.605548 8974 raft_consensus.cc:2781] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:17:07.605641 8974 raft_consensus.cc:482] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:17:07.605675 8974 raft_consensus.cc:3037] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:17:07.606472 8974 raft_consensus.cc:504] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.606621 8974 leader_election.cc:288] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 1 election: Requested vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
I20260620 08:17:07.606856 8903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac1220c67934cb281abea359e7873b3" candidate_uuid: "92fa5034c38043c3a4621b3c60f5962e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06336fae806e4996afa608f118e4d260"
I20260620 08:17:07.606858 8769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac1220c67934cb281abea359e7873b3" candidate_uuid: "92fa5034c38043c3a4621b3c60f5962e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb308b20835493183492d5d75f8d094"
I20260620 08:17:07.606946 8903 raft_consensus.cc:3037] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:17:07.606931 8769 raft_consensus.cc:3037] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:17:07.607654 8903 raft_consensus.cc:2454] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92fa5034c38043c3a4621b3c60f5962e in term 1.
I20260620 08:17:07.607795 8769 raft_consensus.cc:2454] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92fa5034c38043c3a4621b3c60f5962e in term 1.
I20260620 08:17:07.607867 8568 leader_election.cc:302] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06336fae806e4996afa608f118e4d260, 92fa5034c38043c3a4621b3c60f5962e; no voters:
I20260620 08:17:07.607982 8974 raft_consensus.cc:2781] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:17:07.608181 8974 raft_consensus.cc:686] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 1 LEADER]: Becoming Leader. State: Replica: 92fa5034c38043c3a4621b3c60f5962e, State: Running, Role: LEADER
I20260620 08:17:07.608290 8974 consensus_queue.cc:237] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:07.609200 8489 catalog_manager.cc:5495] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129). New cstate: current_term: 1 leader_uuid: "92fa5034c38043c3a4621b3c60f5962e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } health_report { overall_health: HEALTHY } } }
I20260620 08:17:07.658927 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.656250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.24.249.129:45343)
0620 08:17:07.656332 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:07.656334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:07.656343 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:17:07.656393 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:07.656394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:07.656414 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:07.656477 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:17:07.656668 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.657168 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.657950 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:07.658338 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:07.658420 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:07.658569 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:17:07.658676 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:17:07.658740 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:07.658780 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 08:17:07.658798 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:17:07.660828 8903 raft_consensus.cc:1261] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 1 FOLLOWER]: Refusing update from remote peer 92fa5034c38043c3a4621b3c60f5962e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:17:07.660857 8769 raft_consensus.cc:1261] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Refusing update from remote peer 92fa5034c38043c3a4621b3c60f5962e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:17:07.661202 8974 consensus_queue.cc:1035] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [LEADER]: Connected to new peer: Peer: permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:17:07.661356 8977 consensus_queue.cc:1035] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:17:07.663794 8980 mvcc.cc:205] Tried to move back new op lower bound from 7298840279695921152 to 7298840279484157952. Current Snapshot: MvccSnapshot[applied={T|T < 7298840279695921152}]
I20260620 08:17:07.664247 8981 mvcc.cc:205] Tried to move back new op lower bound from 7298840279695921152 to 7298840279484157952. Current Snapshot: MvccSnapshot[applied={T|T < 7298840279695921152}]
I20260620 08:17:07.664992 8982 mvcc.cc:205] Tried to move back new op lower bound from 7298840279695921152 to 7298840279484157952. Current Snapshot: MvccSnapshot[applied={T|T < 7298840279695921152}]
I20260620 08:17:07.666678 8490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:17:07.666829 8490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:17:07.667572 8490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Sending DeleteTablet for 3 replicas of tablet 4ac1220c67934cb281abea359e7873b3
I20260620 08:17:07.667897 8749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac1220c67934cb281abea359e7873b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:17:07 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:41948
I20260620 08:17:07.667902 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac1220c67934cb281abea359e7873b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:17:07 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:48596
I20260620 08:17:07.667999 8883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac1220c67934cb281abea359e7873b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:17:07 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:39012
I20260620 08:17:07.668213 8991 tablet_replica.cc:330] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: stopping tablet replica
I20260620 08:17:07.668202 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 8552
I20260620 08:17:07.668313 8991 raft_consensus.cc:2229] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:17:07.668442 8991 raft_consensus.cc:2258] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:17:07.669288 8991 ts_tablet_manager.cc:1905] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:17:07.671368 8991 ts_tablet_manager.cc:1918] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:17:07.671450 8991 log.cc:1196] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/4ac1220c67934cb281abea359e7873b3
I20260620 08:17:07.671729 8991 ts_tablet_manager.cc:1939] T 4ac1220c67934cb281abea359e7873b3 P 06336fae806e4996afa608f118e4d260: Deleting consensus metadata
I20260620 08:17:07.671972 8989 tablet_replica.cc:330] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: stopping tablet replica
I20260620 08:17:07.672093 8989 raft_consensus.cc:2229] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:17:07.672209 8989 raft_consensus.cc:2258] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:17:07.672224 8474 catalog_manager.cc:4841] TS 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119): tablet 4ac1220c67934cb281abea359e7873b3 (table test-table [id=a07db92d5b994d64944c7512d5a11724]) successfully deleted
I20260620 08:17:07.672610 8989 ts_tablet_manager.cc:1905] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:17:07.674577 8989 ts_tablet_manager.cc:1918] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:17:07.674656 8989 log.cc:1196] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/4ac1220c67934cb281abea359e7873b3
I20260620 08:17:07.674896 8989 ts_tablet_manager.cc:1939] T 4ac1220c67934cb281abea359e7873b3 P 6bb308b20835493183492d5d75f8d094: Deleting consensus metadata
I20260620 08:17:07.675369 8474 catalog_manager.cc:4841] TS 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787): tablet 4ac1220c67934cb281abea359e7873b3 (table test-table [id=a07db92d5b994d64944c7512d5a11724]) successfully deleted
W20260620 08:17:07.676676 8475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:45343 (error 108)
I20260620 08:17:07.676939 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 8686
I20260620 08:17:07.677654 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:07.677139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:45343 (local address 127.0.0.1:48610)
0620 08:17:07.677591 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:07.677625 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
W20260620 08:17:07.677781 8475 catalog_manager.cc:4584] TS 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129:45343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ac1220c67934cb281abea359e7873b3: Network error: Client connection negotiation failed: client connection to 127.24.249.129:45343: connect: Connection refused (error 111)
I20260620 08:17:07.682142 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 8820
I20260620 08:17:07.687090 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 8458
I20260620 08:17:27.693011 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41031
--webserver_interface=127.24.249.190
--webserver_port=44185
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:17:27.778908 8993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:27.779137 8993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:27.779161 8993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:27.780298 8993 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:17:27.780334 8993 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:17:27.780349 8993 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:17:27.780360 8993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:17:27.780375 8993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:27.780388 8993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:27.780401 8993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:17:27.780421 8993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:17:27.781976 8993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:27.782196 8993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:27.782377 8993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:27.792915 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:27.792994 8993 server_base.cc:1034] running on GCE node
W20260620 08:17:27.792949 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:27.792922 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:27.793629 8993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:27.794076 8993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:27.795254 8993 hybrid_clock.cc:648] HybridClock initialized: now 1781943447795243 us; error 35 us; skew 500 ppm
Jun 20 08:17:27 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943447, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:27.798238 8993 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:17:27.798851 8993 webserver.cc:460] Webserver started at http://127.24.249.190:44185/ using document root <none> and password file <none>
I20260620 08:17:27.799147 8993 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:27.799244 8993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:27.800833 8993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:17:27.801715 9009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:27.801996 8993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:27.802117 8993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "7411ba9266644f079b91282af73f2ed0"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "3001c0ee719ea712f80ef9fe2a2ca792"
server_key_iv: "0e5da18418f78dcd8f030c7f42f8f86f"
server_key_version: "encryptionkey@0"
I20260620 08:17:27.802587 8993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:27.819429 8993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:27.822254 8993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:27.822430 8993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:27.827315 8993 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:41031
I20260620 08:17:27.827366 9061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:41031 every 8 connection(s)
I20260620 08:17:27.827798 8993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:17:27.828992 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 8993
I20260620 08:17:27.829322 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:45343
--local_ip_for_outbound_sockets=127.24.249.129
--tserver_master_addrs=127.24.249.190:41031
--webserver_port=44103
--webserver_interface=127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:17:27.829654 9062 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:17:27.830428 9062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Bootstrap starting.
I20260620 08:17:27.842742 9062 log.cc:824] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:27.845458 9062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:17:27.845852 9062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Bootstrap complete.
I20260620 08:17:27.849238 9062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:27.849624 9062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7411ba9266644f079b91282af73f2ed0, State: Initialized, Role: FOLLOWER
I20260620 08:17:27.849814 9062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:27.849929 9062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:17:27.849994 9062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:17:27.850045 9062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 08:17:27.851475 9062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:27.851588 9062 leader_election.cc:302] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7411ba9266644f079b91282af73f2ed0; no voters:
I20260620 08:17:27.851747 9062 leader_election.cc:288] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 08:17:27.851809 9066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 08:17:27.852053 9062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:17:27.852129 9066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 LEADER]: Becoming Leader. State: Replica: 7411ba9266644f079b91282af73f2ed0, State: Running, Role: LEADER
I20260620 08:17:27.852453 9066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:27.852807 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7411ba9266644f079b91282af73f2ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } } }
I20260620 08:17:27.852823 9066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7411ba9266644f079b91282af73f2ed0. Latest consensus state: current_term: 2 leader_uuid: "7411ba9266644f079b91282af73f2ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } } }
I20260620 08:17:27.852990 9066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:17:27.852898 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:17:27.853376 9076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:17:27.854370 9076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:17:27.854530 9076 catalog_manager.cc:1259] Loaded cluster ID: 5d50758508c140c7b43ca3d33d4e11ed
I20260620 08:17:27.854568 9076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:17:27.855710 9076 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:17:27.855932 9076 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Loaded TSK: 0
I20260620 08:17:27.872134 9076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Generated new TSK 1
I20260620 08:17:27.872289 9076 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:17:27.934739 9064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:27.934996 9064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:27.935040 9064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:27.936353 9064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:27.936427 9064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:27.936475 9064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:17:27.938107 9064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:45343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=44103
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:27.938402 9064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:27.938653 9064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:27.944319 9090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:27.944319 9089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:27.944343 9092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:27.945278 9064 server_base.cc:1034] running on GCE node
I20260620 08:17:27.945659 9064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:27.946012 9064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:27.947192 9064 hybrid_clock.cc:648] HybridClock initialized: now 1781943447947170 us; error 41 us; skew 500 ppm
Jun 20 08:17:27 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943447, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:27.950069 9064 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:17:27.950626 9064 webserver.cc:460] Webserver started at http://127.24.249.129:44103/ using document root <none> and password file <none>
I20260620 08:17:27.950946 9064 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:27.951037 9064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:27.952997 9064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:17:27.953926 9099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:27.954169 9064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:27.954267 9064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "92fa5034c38043c3a4621b3c60f5962e"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "891fa6637a8fb14e91714be88cffbc6c"
server_key_iv: "97f106ab6f8d4bdbd3adacc207433ebe"
server_key_version: "encryptionkey@0"
I20260620 08:17:27.954725 9064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:27.967103 9064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:27.969888 9064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:27.970047 9064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:27.970309 9064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:27.970803 9106 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:17:27.972383 9064 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:17:27.972441 9064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 08:17:27.972478 9064 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:17:27.973359 9064 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:17:27.973407 9064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:17:27.973574 9106 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap starting.
I20260620 08:17:27.980707 9064 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:45343
I20260620 08:17:27.981230 9064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:17:27.981487 9213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:45343 every 8 connection(s)
I20260620 08:17:27.984526 9106 log.cc:824] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:27.986220 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 9064
I20260620 08:17:27.986521 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:45787
--local_ip_for_outbound_sockets=127.24.249.130
--tserver_master_addrs=127.24.249.190:41031
--webserver_port=42121
--webserver_interface=127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:17:27.987763 9106 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:17:27.988234 9106 tablet_bootstrap.cc:492] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap complete.
I20260620 08:17:27.989573 9106 ts_tablet_manager.cc:1397] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260620 08:17:27.991135 9106 raft_consensus.cc:348] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:27.991748 9106 raft_consensus.cc:729] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fa5034c38043c3a4621b3c60f5962e, State: Initialized, Role: FOLLOWER
I20260620 08:17:27.991999 9106 consensus_queue.cc:260] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:27.992388 9106 ts_tablet_manager.cc:1428] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Jun 20 08:17:27 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943447, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:17:28.000505 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:27.982251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:39321 (local address 127.24.249.190:41031)
0620 08:17:27.982437 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:27.982446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:27.984978 (+ 2532us) server_negotiation.cc:407] Connection header received
0620 08:17:27.985768 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:27.985776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:27.985835 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:27.986051 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:27.989232 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:27.990651 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:27.992159 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:27.992762 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:27.992847 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:27.995958 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:27.995963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:27.995963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:27.995978 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:27.998235 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:27.998807 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:27.998811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:27.998812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:27.998931 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:27.999254 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:27.999254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:27.999255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:27.999543 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:27.999613 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.000152 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.000289 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260620 08:17:28.000509 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:27.982105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:39321)
0620 08:17:27.984786 (+ 2681us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:27.984833 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:27.985530 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:27.988365 (+ 2835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:27.988370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:27.988772 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:27.989073 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:27.989080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:27.990782 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:27.990783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:27.992039 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:27.992043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:27.992994 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:27.992994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:27.993132 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:27.993826 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:27.993836 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:27.995812 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:27.998330 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:27.998338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:27.998341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:27.998707 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:27.999024 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:27.999025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:27.999025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:27.999181 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:27.999636 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:27.999645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:27.999830 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.000092 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.000289 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2638,"thread_start_us":95,"threads_started":1}
I20260620 08:17:28.001677 9214 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41031
I20260620 08:17:28.001780 9214 heartbeater.cc:464] Registering TS with master...
I20260620 08:17:28.002000 9214 heartbeater.cc:511] Master 127.24.249.190:41031 requested a full tablet report, sending...
I20260620 08:17:28.002804 9026 ts_manager.cc:194] Registered new tserver with Master: 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129:45343)
I20260620 08:17:28.004045 9026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:39321
I20260620 08:17:28.005086 9214 heartbeater.cc:503] Master 127.24.249.190:41031 was elected leader, sending a full tablet report...
Jun 20 08:17:28 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943447, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.129@KRBTEST.COM
I20260620 08:17:28.014691 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.003466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:45343 (local address 127.0.0.1:56336)
0620 08:17:28.004087 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.004126 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:28.004196 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:28.005299 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:28.005306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:28.005657 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:28.005745 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.005747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.007122 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.007123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.008036 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.008039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.008703 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.008704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.008790 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.009360 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:28.009366 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:28.011237 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:28.013371 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:28.013378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:28.013380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:28.013665 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:28.013906 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:28.013907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:28.013907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:28.013975 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:28.014316 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:28.014323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:28.014380 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.014468 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.014589 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":160,"threads_started":1}
I20260620 08:17:28.014766 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.003794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.24.249.129:45343)
0620 08:17:28.004322 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.004327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.004336 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:17:28.004369 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.004372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.004386 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.004498 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:28.005856 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.007010 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.008131 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.008578 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.008640 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.011332 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:28.011340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:28.011341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:28.011348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:28.013281 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.013752 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.013754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.013755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.013818 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.014043 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.014044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.014044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.014227 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:28.014324 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.014565 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.014664 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":178,"threads_started":1}
I20260620 08:17:28.015520 9148 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ac1220c67934cb281abea359e7873b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:17:07 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:56336
I20260620 08:17:28.016160 9223 tablet_replica.cc:330] stopping tablet replica
I20260620 08:17:28.016273 9223 raft_consensus.cc:2229] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:17:28.016342 9223 raft_consensus.cc:2258] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:17:28.016669 9223 ts_tablet_manager.cc:1905] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:17:28.019400 9223 ts_tablet_manager.cc:1918] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:17:28.019479 9223 log.cc:1196] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal/wals/4ac1220c67934cb281abea359e7873b3
I20260620 08:17:28.019742 9223 ts_tablet_manager.cc:1939] T 4ac1220c67934cb281abea359e7873b3 P 92fa5034c38043c3a4621b3c60f5962e: Deleting consensus metadata
I20260620 08:17:28.020323 9011 catalog_manager.cc:4841] TS 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129:45343): tablet 4ac1220c67934cb281abea359e7873b3 (table test-table [id=a07db92d5b994d64944c7512d5a11724]) successfully deleted
W20260620 08:17:28.081043 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:28.081264 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:28.081288 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:28.082443 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:28.082482 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:28.082508 9219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:17:28.083936 9219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:45787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=42121
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:28.084180 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:28.084443 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:28.089781 9229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:28.089790 9228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:28.089790 9231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:28.090108 9219 server_base.cc:1034] running on GCE node
I20260620 08:17:28.090364 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:28.090615 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:28.091775 9219 hybrid_clock.cc:648] HybridClock initialized: now 1781943448091763 us; error 29 us; skew 500 ppm
Jun 20 08:17:28 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943448, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:28.094447 9219 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:17:28.094880 9219 webserver.cc:460] Webserver started at http://127.24.249.130:42121/ using document root <none> and password file <none>
I20260620 08:17:28.095068 9219 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:28.095119 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:28.096446 9219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:17:28.097069 9238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:28.097268 9219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:17:28.097337 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "6bb308b20835493183492d5d75f8d094"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "11e5b52d35c7557a72ede20118df0746"
server_key_iv: "dfa1d701bde3613cf17f05b57b0d9df4"
server_key_version: "encryptionkey@0"
I20260620 08:17:28.097677 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:28.141829 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:28.144580 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:28.144724 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:28.145015 9219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:28.145380 9219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:17:28.145426 9219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:28.145452 9219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:17:28.145468 9219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:28.152148 9219 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:45787
I20260620 08:17:28.152297 9351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:45787 every 8 connection(s)
I20260620 08:17:28.152665 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:17:28.155174 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 9219
I20260620 08:17:28.155467 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:35119
--local_ip_for_outbound_sockets=127.24.249.131
--tserver_master_addrs=127.24.249.190:41031
--webserver_port=33745
--webserver_interface=127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 08:17:28 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943448, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:17:28.163959 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.153604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:41317 (local address 127.24.249.190:41031)
0620 08:17:28.153774 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.153777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.153844 (+ 67us) server_negotiation.cc:407] Connection header received
0620 08:17:28.154415 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.154416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.154434 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.154510 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:28.155756 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.156297 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.157569 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.157799 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.157907 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.160491 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:28.160493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:28.160493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:28.160501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:28.162170 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.162876 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.162878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.162879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.162959 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.163241 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.163242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.163243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.163382 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:28.163464 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.163756 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.163854 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 08:17:28.164127 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.153559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:41317)
0620 08:17:28.153723 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.153766 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:28.154307 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:28.154562 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:28.154566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:28.155297 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:28.155609 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.155619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.156410 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.156411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.157468 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.157471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.157903 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.157904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.158045 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.158614 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:28.158622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:28.160381 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:28.162267 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:28.162275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:28.162277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:28.162757 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:28.163035 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:28.163036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:28.163036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:28.163168 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:28.163456 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:28.163463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:28.163614 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.163721 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.163886 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260620 08:17:28.164616 9352 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41031
I20260620 08:17:28.164783 9352 heartbeater.cc:464] Registering TS with master...
I20260620 08:17:28.165007 9352 heartbeater.cc:511] Master 127.24.249.190:41031 requested a full tablet report, sending...
I20260620 08:17:28.165443 9026 ts_manager.cc:194] Registered new tserver with Master: 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787)
I20260620 08:17:28.165850 9026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:41317
W20260620 08:17:28.244801 9355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:28.245209 9355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:28.245294 9355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:28.246730 9355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:28.246801 9355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:28.246850 9355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:17:28.248425 9355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:35119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=33745
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:28.248718 9355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:28.248963 9355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:28.254396 9361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:28.254396 9360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:28.254401 9363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:28.254760 9355 server_base.cc:1034] running on GCE node
I20260620 08:17:28.254982 9355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:28.255259 9355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:28.256494 9355 hybrid_clock.cc:648] HybridClock initialized: now 1781943448256482 us; error 42 us; skew 500 ppm
Jun 20 08:17:28 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943448, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:28.259299 9355 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:17:28.259858 9355 webserver.cc:460] Webserver started at http://127.24.249.131:33745/ using document root <none> and password file <none>
I20260620 08:17:28.260128 9355 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:28.260213 9355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:28.261749 9355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:17:28.262459 9370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:28.262686 9355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:17:28.262794 9355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "06336fae806e4996afa608f118e4d260"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "74b69a91421add1cad0021ab823d716a"
server_key_iv: "5311e5df2896c7c650f40ea1f0f2b03e"
server_key_version: "encryptionkey@0"
I20260620 08:17:28.263168 9355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:28.276227 9355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:28.278684 9355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:28.278851 9355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:28.279094 9355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:28.279417 9355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:17:28.279472 9355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:28.279515 9355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:17:28.279542 9355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:28.285550 9355 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:35119
I20260620 08:17:28.285595 9483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:35119 every 8 connection(s)
I20260620 08:17:28.286036 9355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:17:28.291604 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 9355
Jun 20 08:17:28 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943448, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:17:28.297577 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.286881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:48521 (local address 127.24.249.190:41031)
0620 08:17:28.286934 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.286936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.287053 (+ 117us) server_negotiation.cc:407] Connection header received
0620 08:17:28.287555 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.287557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.287568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.287642 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:28.288593 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.289118 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.290342 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.290677 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.290765 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.293940 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:28.293942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:28.293942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:28.293950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:28.295972 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.296497 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.296498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.296499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.296560 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.296830 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.296830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.296831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.296952 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:28.296993 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.297393 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.297485 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:17:28.297780 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.286753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:48521)
0620 08:17:28.286912 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.286955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:28.287433 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:28.287696 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:28.287699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:28.288273 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:28.288478 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.288483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.289222 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.289223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.290191 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.290194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.290787 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.290788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.290923 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.291563 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:28.291574 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:28.293776 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:28.296069 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:28.296080 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:28.296082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:28.296404 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:28.296633 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:28.296633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:28.296634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:28.296760 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:28.297039 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:28.297044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:28.297213 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.297350 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.297527 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260620 08:17:28.298641 9484 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41031
I20260620 08:17:28.298754 9484 heartbeater.cc:464] Registering TS with master...
I20260620 08:17:28.298971 9484 heartbeater.cc:511] Master 127.24.249.190:41031 requested a full tablet report, sending...
I20260620 08:17:28.299469 9026 ts_manager.cc:194] Registered new tserver with Master: 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
I20260620 08:17:28.300284 9026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:48521
I20260620 08:17:28.303078 9488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.292064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.24.249.190:41031)
0620 08:17:28.292172 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.292175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.292475 (+ 300us) server_negotiation.cc:407] Connection header received
0620 08:17:28.292527 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.292528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.292542 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.292617 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:28.294976 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.296023 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.297223 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.297561 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.297635 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.299499 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:28.299501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:28.299501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:28.299509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:28.301340 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.301730 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.301731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.301732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.301798 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.302097 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.302097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.302098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.302272 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:28.302688 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.302833 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.302946 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
I20260620 08:17:28.304214 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:17:28.308411 9488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.304642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.24.249.190:41031)
0620 08:17:28.304686 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.304688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.304715 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:17:28.304811 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.304812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.304860 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.304945 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:17:28.305071 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.306083 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.307371 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.307803 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.307890 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.308083 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:17:28.308157 (+ 74us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 08:17:28.308343 (+ 186us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58306: token expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 08:17:28.308530 9488 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58306: token expired
W20260620 08:17:28.308631 8953 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.249.190:41031)
I20260620 08:17:28.316378 9488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.308895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.24.249.190:41031)
0620 08:17:28.308932 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.308934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.308943 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:17:28.309007 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.309008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.309017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.309074 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:28.309621 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.310292 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.311222 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.311533 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.311616 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.313275 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:28.313276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:28.313277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:28.313284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:28.315059 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.315493 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.315493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.315494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.315553 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:28.315777 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:28.315778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:28.315779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:28.315955 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:28.316072 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.316174 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.316276 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:17:28.317739 9026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:17:28.320128 9026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:17:28.322507 9148 tablet_service.cc:1467] Processing CreateTablet for tablet 815e45804a3c416a8a6267ae3c01507c (DEFAULT_TABLE table=test-table [id=4676aa47fac141769450d2c7a628b74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:17:28.322744 9148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815e45804a3c416a8a6267ae3c01507c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:17:28.324630 9106 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap starting.
I20260620 08:17:28.324982 9106 tablet_bootstrap.cc:654] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Neither blocks nor log segments found. Creating new log.
I20260620 08:17:28.325600 9106 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: No bootstrap required, opened a new log
I20260620 08:17:28.325656 9106 ts_tablet_manager.cc:1397] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:28.326095 9106 raft_consensus.cc:348] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.326186 9106 raft_consensus.cc:374] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:17:28.326244 9106 raft_consensus.cc:729] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fa5034c38043c3a4621b3c60f5962e, State: Initialized, Role: FOLLOWER
I20260620 08:17:28.326287 9106 consensus_queue.cc:260] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.326375 9106 ts_tablet_manager.cc:1428] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 08:17:28.327349 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.322221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45787 (local address 127.0.0.1:48720)
0620 08:17:28.322256 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.322268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:28.322319 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:28.323100 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:28.323101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:28.323104 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:28.323157 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.323159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.324269 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.324270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.325496 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.325500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.327032 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.327033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.327121 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.327141 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.327227 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.327275 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 08:17:28.327366 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.322685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.24.249.130:45787)
0620 08:17:28.322879 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.322884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.322896 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:17:28.322938 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.322941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.322958 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.323278 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:28.323407 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.324174 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.325918 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.326933 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.327014 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.327086 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.327246 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.327288 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
I20260620 08:17:28.328642 9286 tablet_service.cc:1467] Processing CreateTablet for tablet 815e45804a3c416a8a6267ae3c01507c (DEFAULT_TABLE table=test-table [id=4676aa47fac141769450d2c7a628b74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:17:28.329095 9286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815e45804a3c416a8a6267ae3c01507c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:17:28.333240 9494 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Bootstrap starting.
I20260620 08:17:28.333948 9494 tablet_bootstrap.cc:654] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Neither blocks nor log segments found. Creating new log.
I20260620 08:17:28.334535 9494 log.cc:824] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:28.335323 9494 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: No bootstrap required, opened a new log
I20260620 08:17:28.335405 9494 ts_tablet_manager.cc:1397] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:17:28.337214 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.322327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35119 (local address 127.0.0.1:38412)
0620 08:17:28.322486 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.322502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:28.322577 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:28.333385 (+ 10808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:28.333386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:28.333388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:28.333445 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.333447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.334824 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.334826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.336033 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.336036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.336995 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.336997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.337062 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.337066 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.337095 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.337135 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":60,"threads_started":1}
I20260620 08:17:28.337255 9494 raft_consensus.cc:348] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.337476 9494 raft_consensus.cc:374] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:17:28.337519 9494 raft_consensus.cc:729] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb308b20835493183492d5d75f8d094, State: Initialized, Role: FOLLOWER
I20260620 08:17:28.337561 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.322916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.24.249.131:35119)
0620 08:17:28.330029 (+ 7113us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.330034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.330047 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:17:28.330089 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.330091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.330109 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.330197 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:28.333545 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.334696 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.336149 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.336895 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.337337 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.337409 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.337435 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.337475 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7062,"thread_start_us":90,"threads_started":1}
I20260620 08:17:28.337652 9494 consensus_queue.cc:260] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.337944 9352 heartbeater.cc:503] Master 127.24.249.190:41031 was elected leader, sending a full tablet report...
I20260620 08:17:28.337941 9494 ts_tablet_manager.cc:1428] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 08:17:28.338296 9418 tablet_service.cc:1467] Processing CreateTablet for tablet 815e45804a3c416a8a6267ae3c01507c (DEFAULT_TABLE table=test-table [id=4676aa47fac141769450d2c7a628b74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:17:28.338622 9418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815e45804a3c416a8a6267ae3c01507c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:17:28.342460 9497 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Bootstrap starting.
I20260620 08:17:28.343235 9497 tablet_bootstrap.cc:654] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Neither blocks nor log segments found. Creating new log.
I20260620 08:17:28.343930 9497 log.cc:824] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:28.344826 9497 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: No bootstrap required, opened a new log
I20260620 08:17:28.344909 9497 ts_tablet_manager.cc:1397] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:17:28.347427 9497 raft_consensus.cc:348] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.347651 9497 raft_consensus.cc:374] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:17:28.347697 9497 raft_consensus.cc:729] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06336fae806e4996afa608f118e4d260, State: Initialized, Role: FOLLOWER
I20260620 08:17:28.347842 9497 consensus_queue.cc:260] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.348171 9484 heartbeater.cc:503] Master 127.24.249.190:41031 was elected leader, sending a full tablet report...
I20260620 08:17:28.348170 9497 ts_tablet_manager.cc:1428] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260620 08:17:28.403362 9353 tablet.cc:2369] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:17:28.482388 9215 tablet.cc:2369] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:17:28.536660 9485 tablet.cc:2369] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:17:28.693318 9499 raft_consensus.cc:482] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:17:28.693456 9499 raft_consensus.cc:504] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.693979 9499 leader_election.cc:288] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129:45343)
I20260620 08:17:28.697122 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.694320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:35367 (local address 127.24.249.130:45787)
0620 08:17:28.694423 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.694426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.694482 (+ 56us) server_negotiation.cc:407] Connection header received
0620 08:17:28.694628 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.694630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.694643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.694717 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:28.694892 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.695378 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.696383 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.696748 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.696845 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.696855 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.697023 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.697064 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 08:17:28.697130 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.694227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45787 (local address 127.24.249.131:35367)
0620 08:17:28.694410 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.694423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:28.694541 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:28.694732 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:28.694733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:28.694735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:28.694822 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.694823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.695469 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.695470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.696283 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.696285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.696846 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.696847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.696925 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.696929 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.696999 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.697041 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":16,"thread_start_us":98,"threads_started":1}
I20260620 08:17:28.697230 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.694227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:45343 (local address 127.24.249.131:56937)
0620 08:17:28.694288 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.694301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:28.694347 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:28.694646 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:28.694648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:28.694650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:17:28.694730 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.694732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.695446 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.695447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.696282 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:28.696284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.696885 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.696886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:28.696976 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.696979 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:17:28.697054 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:17:28.697094 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":32}
I20260620 08:17:28.697186 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.694298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:56937 (local address 127.24.249.129:45343)
0620 08:17:28.694427 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.694431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.694446 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:17:28.694522 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.694523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.694547 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.694640 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:17:28.694804 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.695359 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.696376 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.696789 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.696877 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.696952 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.697061 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.697103 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260620 08:17:28.697474 9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815e45804a3c416a8a6267ae3c01507c" candidate_uuid: "06336fae806e4996afa608f118e4d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb308b20835493183492d5d75f8d094" is_pre_election: true
I20260620 08:17:28.697644 9306 raft_consensus.cc:2454] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06336fae806e4996afa608f118e4d260 in term 0.
I20260620 08:17:28.697674 9168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815e45804a3c416a8a6267ae3c01507c" candidate_uuid: "06336fae806e4996afa608f118e4d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92fa5034c38043c3a4621b3c60f5962e" is_pre_election: true
I20260620 08:17:28.697829 9168 raft_consensus.cc:2454] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06336fae806e4996afa608f118e4d260 in term 0.
I20260620 08:17:28.697929 9371 leader_election.cc:302] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06336fae806e4996afa608f118e4d260, 6bb308b20835493183492d5d75f8d094; no voters:
I20260620 08:17:28.698086 9499 raft_consensus.cc:2781] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:17:28.698149 9499 raft_consensus.cc:482] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:17:28.698171 9499 raft_consensus.cc:3037] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:17:28.698879 9499 raft_consensus.cc:504] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.699030 9499 leader_election.cc:288] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [CANDIDATE]: Term 1 election: Requested vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129:45343)
I20260620 08:17:28.699186 9306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815e45804a3c416a8a6267ae3c01507c" candidate_uuid: "06336fae806e4996afa608f118e4d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb308b20835493183492d5d75f8d094"
I20260620 08:17:28.699218 9168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815e45804a3c416a8a6267ae3c01507c" candidate_uuid: "06336fae806e4996afa608f118e4d260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92fa5034c38043c3a4621b3c60f5962e"
I20260620 08:17:28.699290 9168 raft_consensus.cc:3037] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:17:28.699295 9306 raft_consensus.cc:3037] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:17:28.699992 9168 raft_consensus.cc:2454] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06336fae806e4996afa608f118e4d260 in term 1.
I20260620 08:17:28.700129 9306 raft_consensus.cc:2454] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06336fae806e4996afa608f118e4d260 in term 1.
I20260620 08:17:28.700170 9372 leader_election.cc:302] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06336fae806e4996afa608f118e4d260, 92fa5034c38043c3a4621b3c60f5962e; no voters:
I20260620 08:17:28.700387 9499 raft_consensus.cc:2781] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:17:28.700543 9499 raft_consensus.cc:686] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 1 LEADER]: Becoming Leader. State: Replica: 06336fae806e4996afa608f118e4d260, State: Running, Role: LEADER
I20260620 08:17:28.700654 9499 consensus_queue.cc:237] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:28.701457 9026 catalog_manager.cc:5495] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06336fae806e4996afa608f118e4d260 (127.24.249.131). New cstate: current_term: 1 leader_uuid: "06336fae806e4996afa608f118e4d260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } health_report { overall_health: UNKNOWN } } }
I20260620 08:17:28.773656 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.771306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.24.249.131:35119)
0620 08:17:28.771355 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:28.771356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:28.771363 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:17:28.771428 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:28.771429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:28.771448 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:28.771512 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:17:28.771619 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.772099 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.772839 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:28.773050 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:28.773107 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:28.773246 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:17:28.773348 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:17:28.773448 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:28.773517 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 08:17:28.773542 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:17:28.775583 9306 raft_consensus.cc:1261] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Refusing update from remote peer 06336fae806e4996afa608f118e4d260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:17:28.775609 9168 raft_consensus.cc:1261] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Refusing update from remote peer 06336fae806e4996afa608f118e4d260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:17:28.775985 9499 consensus_queue.cc:1035] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:17:28.776152 9502 consensus_queue.cc:1035] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:17:28.778519 9506 mvcc.cc:205] Tried to move back new op lower bound from 7298840366181855232 to 7298840365878321152. Current Snapshot: MvccSnapshot[applied={T|T < 7298840366181855232}]
I20260620 08:17:28.778645 9507 mvcc.cc:205] Tried to move back new op lower bound from 7298840366181855232 to 7298840365878321152. Current Snapshot: MvccSnapshot[applied={T|T < 7298840366181855232}]
I20260620 08:17:28.780720 9025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:17:28.780802 9025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:17:28.781585 9025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Sending DeleteTablet for 3 replicas of tablet 815e45804a3c416a8a6267ae3c01507c
I20260620 08:17:28.781880 9286 tablet_service.cc:1514] Processing DeleteTablet for tablet 815e45804a3c416a8a6267ae3c01507c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:17:28 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:48720
I20260620 08:17:28.781927 9418 tablet_service.cc:1514] Processing DeleteTablet for tablet 815e45804a3c416a8a6267ae3c01507c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:17:28 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:38412
I20260620 08:17:28.782011 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 9064
I20260620 08:17:28.782135 9516 tablet_replica.cc:330] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: stopping tablet replica
I20260620 08:17:28.782276 9516 raft_consensus.cc:2229] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:17:28.782455 9516 raft_consensus.cc:2258] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:17:28.782970 9516 ts_tablet_manager.cc:1905] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:17:28.785120 9516 ts_tablet_manager.cc:1918] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:17:28.785256 9516 log.cc:1196] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/815e45804a3c416a8a6267ae3c01507c
I20260620 08:17:28.785616 9516 ts_tablet_manager.cc:1939] T 815e45804a3c416a8a6267ae3c01507c P 06336fae806e4996afa608f118e4d260: Deleting consensus metadata
I20260620 08:17:28.787801 9010 catalog_manager.cc:4841] TS 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119): tablet 815e45804a3c416a8a6267ae3c01507c (table test-table [id=4676aa47fac141769450d2c7a628b74c]) successfully deleted
I20260620 08:17:28.790046 9515 tablet_replica.cc:330] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: stopping tablet replica
I20260620 08:17:28.790205 9515 raft_consensus.cc:2229] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:17:28.790349 9515 raft_consensus.cc:2258] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:17:28.790995 9515 ts_tablet_manager.cc:1905] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:17:28.791889 9011 connection.cc:535] client connection to 127.24.249.129:45343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 08:17:28.792022 9011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 08:17:28.792481 9490 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:17:28.792360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:45343 (local address 127.0.0.1:56348)
0620 08:17:28.792405 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.792444 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:45343: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 08:17:28.792773 9515 ts_tablet_manager.cc:1918] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:17:28.792852 9515 log.cc:1196] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/815e45804a3c416a8a6267ae3c01507c
I20260620 08:17:28.792872 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 9219
I20260620 08:17:28.793124 9515 ts_tablet_manager.cc:1939] T 815e45804a3c416a8a6267ae3c01507c P 6bb308b20835493183492d5d75f8d094: Deleting consensus metadata
W20260620 08:17:28.792780 9011 catalog_manager.cc:4584] TS 92fa5034c38043c3a4621b3c60f5962e (127.24.249.129:45343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 815e45804a3c416a8a6267ae3c01507c: Network error: Client connection negotiation failed: client connection to 127.24.249.129:45343: connect: Connection reset by peer (error 104)
I20260620 08:17:28.800066 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 9355
I20260620 08:17:28.800227 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:28.800076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45787 (local address 127.0.0.1:48732)
0620 08:17:28.800137 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:28.800166 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.130:45787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 08:17:28.805516 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 8993
I20260620 08:17:48.811335 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41031
--webserver_interface=127.24.249.190
--webserver_port=44185
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:17:48.893914 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:48.894119 9518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:48.894142 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:48.895272 9518 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 08:17:48.895311 9518 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 08:17:48.895325 9518 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 08:17:48.895337 9518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:17:48.895349 9518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:48.895363 9518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:48.895375 9518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:17:48.895387 9518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:17:48.896916 9518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=44185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:48.897126 9518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:48.897302 9518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:48.902432 9524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:48.902617 9518 server_base.cc:1034] running on GCE node
W20260620 08:17:48.902441 9523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:48.902441 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:48.903005 9518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:48.903370 9518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:48.904531 9518 hybrid_clock.cc:648] HybridClock initialized: now 1781943468904519 us; error 67 us; skew 500 ppm
Jun 20 08:17:48 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943468, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:48.907531 9518 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:17:48.908051 9518 webserver.cc:460] Webserver started at http://127.24.249.190:44185/ using document root <none> and password file <none>
I20260620 08:17:48.908286 9518 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:48.908334 9518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:48.909802 9518 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:17:48.912045 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:48.912279 9518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:17:48.912374 9518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "7411ba9266644f079b91282af73f2ed0"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "3001c0ee719ea712f80ef9fe2a2ca792"
server_key_iv: "0e5da18418f78dcd8f030c7f42f8f86f"
server_key_version: "encryptionkey@0"
I20260620 08:17:48.912807 9518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:48.927089 9518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:48.929689 9518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:48.929809 9518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:48.933647 9518 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:41031
I20260620 08:17:48.933724 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:41031 every 8 connection(s)
I20260620 08:17:48.934122 9518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:17:48.935730 9586 sys_catalog.cc:263] Verifying existing consensus state
I20260620 08:17:48.936439 9586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Bootstrap starting.
I20260620 08:17:48.936985 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 9518
I20260620 08:17:48.937314 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:45343
--local_ip_for_outbound_sockets=127.24.249.129
--tserver_master_addrs=127.24.249.190:41031
--webserver_port=44103
--webserver_interface=127.24.249.129
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:17:48.946118 9586 log.cc:824] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:48.948443 9586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:17:48.948774 9586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Bootstrap complete.
I20260620 08:17:48.951224 9586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:48.951473 9586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7411ba9266644f079b91282af73f2ed0, State: Initialized, Role: FOLLOWER
I20260620 08:17:48.951659 9586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:48.951737 9586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:17:48.951790 9586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:17:48.951839 9586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 08:17:48.952893 9586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:48.953012 9586 leader_election.cc:302] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7411ba9266644f079b91282af73f2ed0; no voters:
I20260620 08:17:48.953207 9586 leader_election.cc:288] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 08:17:48.953277 9590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 08:17:48.953437 9590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [term 3 LEADER]: Becoming Leader. State: Replica: 7411ba9266644f079b91282af73f2ed0, State: Running, Role: LEADER
I20260620 08:17:48.953480 9586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:17:48.953542 9590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } }
I20260620 08:17:48.953783 9590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7411ba9266644f079b91282af73f2ed0. Latest consensus state: current_term: 3 leader_uuid: "7411ba9266644f079b91282af73f2ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } } }
I20260620 08:17:48.953902 9590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:17:48.953783 9591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7411ba9266644f079b91282af73f2ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7411ba9266644f079b91282af73f2ed0" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41031 } } }
I20260620 08:17:48.954092 9591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0 [sys.catalog]: This master's current role is: LEADER
I20260620 08:17:48.954097 9596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:17:48.954949 9596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:17:48.955144 9596 catalog_manager.cc:1259] Loaded cluster ID: 5d50758508c140c7b43ca3d33d4e11ed
I20260620 08:17:48.955261 9596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:17:48.956452 9596 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:17:48.956624 9596 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Loaded TSK: 0
I20260620 08:17:48.956669 9596 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7411ba9266644f079b91282af73f2ed0: Loaded TSK: 1
F20260620 08:17:48.956880 9596 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f46c7d909ad google::LogMessage::Fail() at ??:0
@ 0x7f46c7d9388a google::LogMessage::SendToLog() at ??:0
@ 0x7f46c7d904de google::LogMessage::Flush() at ??:0
@ 0x7f46c7d91a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f46ca4fd104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f46ca4fad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f46cfcea1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f46cfcea48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f46cfcea594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f46cfcb0c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f46cfcbaad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f46c9c5e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f46c9c53954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f46caab76db start_thread at ??:0
@ 0x7f46c688e61f clone at ??:0
W20260620 08:17:49.041252 9588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:49.041486 9588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:49.041525 9588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:49.042805 9588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:49.042869 9588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:49.042922 9588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:17:49.044397 9588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:45343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=44103
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:49.044656 9588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:49.044919 9588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:49.050381 9628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:49.050501 9630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:49.050402 9627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:49.051178 9588 server_base.cc:1034] running on GCE node
I20260620 08:17:49.051401 9588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:49.051674 9588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:49.052834 9588 hybrid_clock.cc:648] HybridClock initialized: now 1781943469052816 us; error 36 us; skew 500 ppm
Jun 20 08:17:49 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:49.055589 9588 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:17:49.056032 9588 webserver.cc:460] Webserver started at http://127.24.249.129:44103/ using document root <none> and password file <none>
I20260620 08:17:49.056270 9588 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:49.056341 9588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:49.057785 9588 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:49.058636 9637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:49.058880 9588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:49.059002 9588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "92fa5034c38043c3a4621b3c60f5962e"
format_stamp: "Formatted at 2026-06-20 08:17:06 on dist-test-slave-65w8"
server_key: "891fa6637a8fb14e91714be88cffbc6c"
server_key_iv: "97f106ab6f8d4bdbd3adacc207433ebe"
server_key_version: "encryptionkey@0"
I20260620 08:17:49.059403 9588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:49.068899 9588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:49.071470 9588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:49.071656 9588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:49.071898 9588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:49.072389 9644 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 08:17:49.074502 9588 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 08:17:49.074616 9588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 08:17:49.074667 9588 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 08:17:49.075347 9588 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 08:17:49.075408 9588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 08:17:49.075485 9644 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap starting.
I20260620 08:17:49.083328 9588 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:45343
I20260620 08:17:49.083506 9751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:45343 every 8 connection(s)
I20260620 08:17:49.083802 9588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:17:49.086934 9644 log.cc:824] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:17:49.089797 9644 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 08:17:49.090174 9644 tablet_bootstrap.cc:492] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Bootstrap complete.
I20260620 08:17:49.091539 9644 ts_tablet_manager.cc:1397] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260620 08:17:49.093084 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 9588
I20260620 08:17:49.093387 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:45787
--local_ip_for_outbound_sockets=127.24.249.130
--tserver_master_addrs=127.24.249.190:41031
--webserver_port=42121
--webserver_interface=127.24.249.130
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:17:49.093144 9644 raft_consensus.cc:348] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:49.093487 9644 raft_consensus.cc:729] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fa5034c38043c3a4621b3c60f5962e, State: Initialized, Role: FOLLOWER
I20260620 08:17:49.093665 9644 consensus_queue.cc:260] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:49.093979 9644 ts_tablet_manager.cc:1428] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20260620 08:17:49.191335 9754 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:17:49.084506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:47043)
0620 08:17:49.084680 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.084724 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:49.085196 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:49.191111 (+105915us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
W20260620 08:17:49.191636 9752 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:41031 (0 consecutive failures): Network error: Failed to ping master at 127.24.249.190:41031: Client connection negotiation failed: client connection to 127.24.249.190:41031: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 08:17:49.192034 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.191882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:52327)
0620 08:17:49.191939 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.192011 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:17:49.192497 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.192370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:50469)
0620 08:17:49.192444 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.192475 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 08:17:49.192672 9752 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 08:17:49.200131 9757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:49.200335 9757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:49.200362 9757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:49.201462 9757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:49.201493 9757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:49.201517 9757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:17:49.202987 9757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:45787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=42121
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:49.203181 9757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:49.203383 9757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:49.208630 9765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:49.208630 9762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:49.208655 9763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:49.209285 9757 server_base.cc:1034] running on GCE node
I20260620 08:17:49.209692 9757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:49.210017 9757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:49.211202 9757 hybrid_clock.cc:648] HybridClock initialized: now 1781943469211198 us; error 40 us; skew 500 ppm
Jun 20 08:17:49 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:49.213979 9757 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:17:49.214485 9757 webserver.cc:460] Webserver started at http://127.24.249.130:42121/ using document root <none> and password file <none>
I20260620 08:17:49.214691 9757 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:49.214735 9757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:49.216120 9757 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:49.217016 9772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:49.217252 9757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:49.217358 9757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "6bb308b20835493183492d5d75f8d094"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "11e5b52d35c7557a72ede20118df0746"
server_key_iv: "dfa1d701bde3613cf17f05b57b0d9df4"
server_key_version: "encryptionkey@0"
I20260620 08:17:49.217777 9757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:49.240971 9757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:49.243486 9757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:49.243638 9757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:49.243908 9757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:49.244240 9757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:17:49.244297 9757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:49.244339 9757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:17:49.244367 9757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:49.250756 9757 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:45787
I20260620 08:17:49.250808 9885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:45787 every 8 connection(s)
I20260620 08:17:49.251154 9757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:17:49.252336 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.251906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:55063)
0620 08:17:49.252116 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.252193 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
W20260620 08:17:49.252908 9886 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:41031 (0 consecutive failures): Network error: Failed to ping master at 127.24.249.190:41031: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
I20260620 08:17:49.253454 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.253127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:56957)
0620 08:17:49.253397 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.253435 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260620 08:17:49.253909 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.253766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:35067)
0620 08:17:49.253816 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.253862 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:17:49.254033 9886 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:17:49.260099 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 9757
I20260620 08:17:49.260353 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:35119
--local_ip_for_outbound_sockets=127.24.249.131
--tserver_master_addrs=127.24.249.190:41031
--webserver_port=33745
--webserver_interface=127.24.249.131
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:17:49.341539 9889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:17:49.341753 9889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:17:49.341776 9889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:17:49.342911 9889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:17:49.342962 9889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:17:49.342988 9889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:17:49.344415 9889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:35119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=33745
--tserver_master_addrs=127.24.249.190:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:17:49.344614 9889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:17:49.344784 9889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:17:49.350168 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:49.350199 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:17:49.350168 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:17:49.350845 9889 server_base.cc:1034] running on GCE node
I20260620 08:17:49.351091 9889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:17:49.351395 9889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:17:49.352584 9889 hybrid_clock.cc:648] HybridClock initialized: now 1781943469352579 us; error 41 us; skew 500 ppm
Jun 20 08:17:49 dist-test-slave-65w8 krb5kdc[8445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:17:49.355242 9889 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:17:49.355746 9889 webserver.cc:460] Webserver started at http://127.24.249.131:33745/ using document root <none> and password file <none>
I20260620 08:17:49.356024 9889 fs_manager.cc:359] Metadata directory not provided
I20260620 08:17:49.356096 9889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:17:49.357468 9889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:49.358177 9904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:49.358413 9889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:17:49.358518 9889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "06336fae806e4996afa608f118e4d260"
format_stamp: "Formatted at 2026-06-20 08:17:07 on dist-test-slave-65w8"
server_key: "74b69a91421add1cad0021ab823d716a"
server_key_iv: "5311e5df2896c7c650f40ea1f0f2b03e"
server_key_version: "encryptionkey@0"
I20260620 08:17:49.358878 9889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:17:49.408067 9889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:17:49.410210 9889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:17:49.410383 9889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:17:49.410666 9889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:17:49.411000 9889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:17:49.411056 9889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:49.411098 9889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:17:49.411127 9889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:17:49.417562 9889 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:35119
I20260620 08:17:49.417605 10017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:35119 every 8 connection(s)
I20260620 08:17:49.418016 9889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:17:49.419137 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.418717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:51955)
0620 08:17:49.418910 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.418992 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20260620 08:17:49.419612 10018 heartbeater.cc:644] Failed to heartbeat to 127.24.249.190:41031 (0 consecutive failures): Network error: Failed to ping master at 127.24.249.190:41031: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
I20260620 08:17:49.419902 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.419811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:40499)
0620 08:17:49.419855 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.419883 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 08:17:49.420166 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:49.420112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:51171)
0620 08:17:49.420135 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:49.420154 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 08:17:49.420280 10018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 08:17:49.426585 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 9889
I20260620 08:17:50.193356 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:50.193112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:44007)
0620 08:17:50.193251 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:50.193293 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260620 08:17:50.254578 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:50.254351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:43287)
0620 08:17:50.254469 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:50.254523 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260620 08:17:50.421000 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:50.420768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:35513)
0620 08:17:50.420901 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:50.420937 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260620 08:17:50.470932 10025 raft_consensus.cc:482] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:17:50.471063 10025 raft_consensus.cc:504] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:50.471379 10025 leader_election.cc:288] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
Jun 20 08:17:50 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.130@KRBTEST.COM
Jun 20 08:17:50 dist-test-slave-65w8 krb5kdc[8445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943469, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.131@KRBTEST.COM
I20260620 08:17:50.484223 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:50.471741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:35119 (local address 127.24.249.129:34053)
0620 08:17:50.471879 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:50.471888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:50.471925 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:50.472718 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:50.472719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:50.473047 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:50.473231 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:50.473236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.475015 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.475016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:50.475816 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:50.475819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.476452 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.476453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:50.476568 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:50.477401 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:50.477406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:50.479709 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:50.482095 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:50.482096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:50.482096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:50.482357 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:50.483048 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:50.483049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:50.483049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:50.483122 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:50.483660 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:50.483661 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:50.483716 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 08:17:50.483816 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 08:17:50.484048 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":57,"thread_start_us":63,"threads_started":1}
I20260620 08:17:50.484252 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:50.471821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:34053 (local address 127.24.249.131:35119)
0620 08:17:50.471967 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:50.471975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:50.471999 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:17:50.472429 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:50.472431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:50.472473 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:50.472617 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:50.473386 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.474866 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.475918 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.476346 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.476482 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:50.479832 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:50.479837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:50.479837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:50.479848 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:50.482006 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:50.482464 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:50.482475 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:50.482476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:50.482800 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:50.483220 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:50.483221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:50.483222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:50.483562 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:50.483658 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:50.483973 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 08:17:50.484131 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":50,"threads_started":1}
I20260620 08:17:50.484275 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:50.471708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:36485 (local address 127.24.249.130:45787)
0620 08:17:50.471921 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 08:17:50.471935 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 08:17:50.471971 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:17:50.472429 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:17:50.472431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:17:50.472473 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:17:50.472627 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:17:50.473424 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.474866 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.476013 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.476415 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.476553 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:50.479583 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:17:50.479588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:17:50.479588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:17:50.479599 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:17:50.481604 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:50.482168 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:50.482171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:50.482171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:50.482291 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:17:50.483048 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:17:50.483049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:17:50.483049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:17:50.483407 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:17:50.483547 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 08:17:50.484019 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 08:17:50.484159 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260620 08:17:50.484288 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:50.471565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:45787 (local address 127.24.249.129:36485)
0620 08:17:50.471682 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:50.471698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 08:17:50.471775 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:17:50.472666 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:17:50.472669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:17:50.473068 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:17:50.473231 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:50.473236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.475004 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.475005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:50.475888 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:17:50.475891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:17:50.476515 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:17:50.476517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:17:50.476572 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:17:50.477306 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:17:50.477325 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:17:50.479348 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:17:50.481713 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:50.481722 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:50.481725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:50.482082 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:50.482454 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:17:50.482455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:17:50.482456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:17:50.482839 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:17:50.483493 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:17:50.483500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:17:50.483666 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 08:17:50.483918 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 08:17:50.484133 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":126}
W20260620 08:17:50.485457 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
W20260620 08:17:50.485633 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
I20260620 08:17:50.485699 9638 leader_election.cc:302] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fa5034c38043c3a4621b3c60f5962e; no voters: 06336fae806e4996afa608f118e4d260, 6bb308b20835493183492d5d75f8d094
I20260620 08:17:50.485831 10025 raft_consensus.cc:2726] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:17:51.194262 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:51.194035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:34137)
0620 08:17:51.194174 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:51.194214 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260620 08:17:51.255425 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:51.255109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:53551)
0620 08:17:51.255277 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:51.255332 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260620 08:17:51.421797 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:51.421525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:46461)
0620 08:17:51.421707 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:51.421756 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260620 08:17:52.195170 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:52.194910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:37561)
0620 08:17:52.195081 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:52.195123 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260620 08:17:52.256125 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:52.255892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:35335)
0620 08:17:52.256043 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:52.256081 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260620 08:17:52.422648 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:52.422388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:47717)
0620 08:17:52.422568 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:52.422604 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
I20260620 08:17:52.647835 10035 raft_consensus.cc:482] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:17:52.647935 10035 raft_consensus.cc:504] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:52.648131 10035 leader_election.cc:288] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
W20260620 08:17:52.648739 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
W20260620 08:17:52.648828 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
I20260620 08:17:52.648862 9638 leader_election.cc:302] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fa5034c38043c3a4621b3c60f5962e; no voters: 06336fae806e4996afa608f118e4d260, 6bb308b20835493183492d5d75f8d094
I20260620 08:17:52.649020 10035 raft_consensus.cc:2726] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:17:53.196092 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:53.195825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:60427)
0620 08:17:53.195999 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:53.196045 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260620 08:17:53.256927 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:53.256667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:37969)
0620 08:17:53.256836 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:53.256882 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260620 08:17:53.423380 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:53.423167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:57555)
0620 08:17:53.423296 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:53.423334 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260620 08:17:54.196910 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:54.196690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:57293)
0620 08:17:54.196825 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:54.196865 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260620 08:17:54.257767 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:54.257518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:45953)
0620 08:17:54.257673 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:54.257714 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260620 08:17:54.424186 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:54.423941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:33647)
0620 08:17:54.424100 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:54.424140 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260620 08:17:55.197738 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:55.197506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:40439)
0620 08:17:55.197654 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:55.197692 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260620 08:17:55.258548 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:55.258292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:32963)
0620 08:17:55.258463 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:55.258504 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260620 08:17:55.425045 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:55.424785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:39243)
0620 08:17:55.424946 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:55.424990 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260620 08:17:55.911337 10045 raft_consensus.cc:482] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:17:55.911448 10045 raft_consensus.cc:504] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:55.911824 10045 leader_election.cc:288] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
W20260620 08:17:55.912302 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
W20260620 08:17:55.912420 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
I20260620 08:17:55.912449 9638 leader_election.cc:302] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fa5034c38043c3a4621b3c60f5962e; no voters: 06336fae806e4996afa608f118e4d260, 6bb308b20835493183492d5d75f8d094
I20260620 08:17:55.912551 10045 raft_consensus.cc:2726] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:17:56.198514 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:56.198245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:52665)
0620 08:17:56.198428 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:56.198467 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":65,"threads_started":1}
I20260620 08:17:56.259370 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:56.259148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:44595)
0620 08:17:56.259286 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:56.259324 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260620 08:17:56.425951 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:56.425645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:35445)
0620 08:17:56.425808 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:56.425848 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260620 08:17:57.199263 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:57.199021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:34747)
0620 08:17:57.199176 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:57.199215 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260620 08:17:57.260171 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:57.259915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:38477)
0620 08:17:57.260076 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:57.260123 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260620 08:17:57.426748 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:57.426495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:40291)
0620 08:17:57.426656 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:57.426699 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260620 08:17:58.200134 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:58.199889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:50951)
0620 08:17:58.200042 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:58.200086 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
I20260620 08:17:58.261010 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:58.260748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:45201)
0620 08:17:58.260904 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:58.260944 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260620 08:17:58.427562 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:58.427255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:58715)
0620 08:17:58.427472 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:58.427516 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":60,"threads_started":1}
I20260620 08:17:58.578872 10055 raft_consensus.cc:482] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:17:58.578971 10055 raft_consensus.cc:504] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:17:58.579105 10055 leader_election.cc:288] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
W20260620 08:17:58.579559 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
W20260620 08:17:58.579644 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
I20260620 08:17:58.579664 9638 leader_election.cc:302] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fa5034c38043c3a4621b3c60f5962e; no voters: 06336fae806e4996afa608f118e4d260, 6bb308b20835493183492d5d75f8d094
I20260620 08:17:58.579841 10055 raft_consensus.cc:2726] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:17:59.200959 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:59.200742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:53771)
0620 08:17:59.200874 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:59.200912 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260620 08:17:59.261755 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:59.261513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:38853)
0620 08:17:59.261640 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:59.261702 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260620 08:17:59.428433 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:17:59.428174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:55843)
0620 08:17:59.428303 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 08:17:59.428346 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260620 08:18:00.201923 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:00.201642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:42995)
0620 08:18:00.201780 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:00.201822 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260620 08:18:00.262486 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:00.262226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:36737)
0620 08:18:00.262356 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:00.262398 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260620 08:18:00.429265 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:00.428999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:33843)
0620 08:18:00.429148 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:00.429182 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260620 08:18:01.202764 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:01.202437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:54979)
0620 08:18:01.202686 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:01.202717 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":187,"threads_started":1}
I20260620 08:18:01.263237 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:01.263015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:38171)
0620 08:18:01.263154 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:01.263189 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260620 08:18:01.430222 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:01.429968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:38903)
0620 08:18:01.430129 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:01.430172 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260620 08:18:02.203560 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:02.203315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:49179)
0620 08:18:02.203464 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:02.203513 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260620 08:18:02.264109 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:02.263855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:57319)
0620 08:18:02.264008 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:02.264052 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260620 08:18:02.431020 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:02.430763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:46079)
0620 08:18:02.430921 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:02.430966 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260620 08:18:03.204408 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:03.204141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:59661)
0620 08:18:03.204311 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:03.204358 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260620 08:18:03.264883 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:03.264627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:35963)
0620 08:18:03.264787 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:03.264827 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260620 08:18:03.432009 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:03.431710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:48809)
0620 08:18:03.431893 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:03.431933 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
I20260620 08:18:04.205300 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:04.205047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.129:55571)
0620 08:18:04.205213 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:04.205255 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
I20260620 08:18:04.242091 10072 raft_consensus.cc:482] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:18:04.242221 10072 raft_consensus.cc:504] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bb308b20835493183492d5d75f8d094" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 45787 } } peers { permanent_uuid: "06336fae806e4996afa608f118e4d260" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 35119 } } peers { permanent_uuid: "92fa5034c38043c3a4621b3c60f5962e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 45343 } }
I20260620 08:18:04.242445 10072 leader_election.cc:288] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787), 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119)
W20260620 08:18:04.242944 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6bb308b20835493183492d5d75f8d094 (127.24.249.130:45787): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
W20260620 08:18:04.243017 9638 leader_election.cc:341] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 06336fae806e4996afa608f118e4d260 (127.24.249.131:35119): Not found: Tablet not found: 815e45804a3c416a8a6267ae3c01507c
I20260620 08:18:04.243068 9638 leader_election.cc:302] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92fa5034c38043c3a4621b3c60f5962e; no voters: 06336fae806e4996afa608f118e4d260, 6bb308b20835493183492d5d75f8d094
I20260620 08:18:04.243161 10072 raft_consensus.cc:2726] T 815e45804a3c416a8a6267ae3c01507c P 92fa5034c38043c3a4621b3c60f5962e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 08:18:04.265668 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:04.265433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.130:47663)
0620 08:18:04.265589 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:04.265624 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260620 08:18:04.432061 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 9588
I20260620 08:18:04.433950 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:04.433671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41031 (local address 127.24.249.131:33467)
0620 08:18:04.433819 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:04.433900 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.190:41031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":57,"threads_started":1}
I20260620 08:18:04.437490 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 9757
I20260620 08:18:04.442116 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 9889
2026-06-20T08:18:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58995 ms)
[----------] 30 tests from SecurityITest (240778 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:04 dist-test-slave-65w8 krb5kdc[10077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:04 dist-test-slave-65w8 krb5kdc[10077](info): set up 2 sockets
Jun 20 08:18:04 dist-test-slave-65w8 krb5kdc[10077](info): commencing operation
krb5kdc: starting...
W20260620 08:18:05.509727 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:05 dist-test-slave-65w8 krb5kdc[10077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:05.639072 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42357
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:42357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:05.721743 10090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:05.722045 10090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:05.722088 10090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:05.723285 10090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:05.723364 10090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:05.723390 10090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:05.723420 10090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:05.723448 10090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:05.724927 10090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:42357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:05.725201 10090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:05.725435 10090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:05.730587 10098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:05.730618 10090 server_base.cc:1034] running on GCE node
W20260620 08:18:05.730650 10096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:05.730588 10095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:05.731148 10090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:05.731416 10090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:05.732581 10090 hybrid_clock.cc:648] HybridClock initialized: now 1781943485732567 us; error 36 us; skew 500 ppm
Jun 20 08:18:05 dist-test-slave-65w8 krb5kdc[10077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943485, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:05.735221 10090 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:05.735756 10090 webserver.cc:460] Webserver started at http://127.24.249.190:40321/ using document root <none> and password file <none>
I20260620 08:18:05.735994 10090 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:05.736063 10090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:05.736227 10090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:05.737332 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "3bdaa8bb85614e33a137ae21308a72d2"
format_stamp: "Formatted at 2026-06-20 08:18:05 on dist-test-slave-65w8"
server_key: "0922db9478b003f7387827039ee1efc6"
server_key_iv: "915b21ff40467751e03f593e3a3230c2"
server_key_version: "encryptionkey@0"
I20260620 08:18:05.737716 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "3bdaa8bb85614e33a137ae21308a72d2"
format_stamp: "Formatted at 2026-06-20 08:18:05 on dist-test-slave-65w8"
server_key: "0922db9478b003f7387827039ee1efc6"
server_key_iv: "915b21ff40467751e03f593e3a3230c2"
server_key_version: "encryptionkey@0"
I20260620 08:18:05.739193 10090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:05.740105 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:05.740334 10090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:05.740448 10090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "3bdaa8bb85614e33a137ae21308a72d2"
format_stamp: "Formatted at 2026-06-20 08:18:05 on dist-test-slave-65w8"
server_key: "0922db9478b003f7387827039ee1efc6"
server_key_iv: "915b21ff40467751e03f593e3a3230c2"
server_key_version: "encryptionkey@0"
I20260620 08:18:05.740525 10090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:05.767115 10090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:05.769737 10090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:05.769961 10090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:05.774434 10090 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:42357
I20260620 08:18:05.774497 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:42357 every 8 connection(s)
I20260620 08:18:05.774909 10090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:05.776042 10158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:05.778398 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2: Bootstrap starting.
I20260620 08:18:05.778935 10158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:05.779191 10158 log.cc:824] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:05.780071 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2: No bootstrap required, opened a new log
I20260620 08:18:05.781682 10158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdaa8bb85614e33a137ae21308a72d2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42357 } }
I20260620 08:18:05.781827 10158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:05.781852 10158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bdaa8bb85614e33a137ae21308a72d2, State: Initialized, Role: FOLLOWER
I20260620 08:18:05.782047 10158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdaa8bb85614e33a137ae21308a72d2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42357 } }
I20260620 08:18:05.782117 10158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:05.782140 10158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:05.782168 10158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:05.782712 10158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdaa8bb85614e33a137ae21308a72d2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42357 } }
I20260620 08:18:05.782799 10158 leader_election.cc:302] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bdaa8bb85614e33a137ae21308a72d2; no voters:
I20260620 08:18:05.782935 10158 leader_election.cc:288] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:05.783011 10161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:05.783382 10161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [term 1 LEADER]: Becoming Leader. State: Replica: 3bdaa8bb85614e33a137ae21308a72d2, State: Running, Role: LEADER
I20260620 08:18:05.783453 10158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:05.783519 10161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdaa8bb85614e33a137ae21308a72d2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42357 } }
I20260620 08:18:05.783984 10162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bdaa8bb85614e33a137ae21308a72d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdaa8bb85614e33a137ae21308a72d2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42357 } } }
I20260620 08:18:05.784123 10163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bdaa8bb85614e33a137ae21308a72d2. Latest consensus state: current_term: 1 leader_uuid: "3bdaa8bb85614e33a137ae21308a72d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bdaa8bb85614e33a137ae21308a72d2" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42357 } } }
I20260620 08:18:05.784189 10162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:05.784215 10163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:05.784716 10171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:05.785027 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10090
I20260620 08:18:05.785153 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:05.785292 10171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:05.785391 25574 external_mini_cluster.cc:1428] Setting key 2308f1be529a29dd12520d29b4cbc5ec
I20260620 08:18:05.787094 10171 catalog_manager.cc:1347] Generated new cluster ID: 59da58767b3442dcbc353b3582352f4c
I20260620 08:18:05.787155 10171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:18:05 dist-test-slave-65w8 krb5kdc[10077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:05.797667 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:05.786297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.24.249.190:42357)
0620 08:18:05.786753 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:05.786762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:05.786798 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:18:05.786896 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:05.786902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:05.786948 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:05.787131 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:05.787831 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:05.789014 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:05.790349 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:05.790795 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:05.791016 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:05.793746 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:05.793754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:05.793754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:05.793771 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:05.795834 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:05.796377 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:05.796380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:05.796380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:05.796537 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:05.796771 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:05.796772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:05.796772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:05.797031 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:05.797210 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:05.797282 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 08:18:05.797437 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":98,"threads_started":1}
I20260620 08:18:05.802217 10171 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:05.802940 10171 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:05.812363 10171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2: Generated new TSK 0
I20260620 08:18:05.812584 10171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:18:05.845395 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42357
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:05.934849 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:05.935052 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:05.935076 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:05.936237 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:05.936278 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:05.936304 10184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:18:05.937744 10184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:05.938020 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:05.938205 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:05.943677 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:05.943755 10184 server_base.cc:1034] running on GCE node
W20260620 08:18:05.943682 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:05.943682 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:05.944214 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:05.944486 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:05.945685 10184 hybrid_clock.cc:648] HybridClock initialized: now 1781943485945670 us; error 32 us; skew 500 ppm
Jun 20 08:18:05 dist-test-slave-65w8 krb5kdc[10077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943485, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:05.948333 10184 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:18:05.948844 10184 webserver.cc:460] Webserver started at http://127.24.249.129:38991/ using document root <none> and password file <none>
I20260620 08:18:05.949088 10184 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:05.949157 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:05.949343 10184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:05.950457 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "d8faedd5a03d40fcb6414a889e2dfb3e"
format_stamp: "Formatted at 2026-06-20 08:18:05 on dist-test-slave-65w8"
server_key: "0757812e8d2ec23ba02361d1fa2ca06e"
server_key_iv: "17ef343c2a49960e3b35e9c8739a1c1a"
server_key_version: "encryptionkey@0"
I20260620 08:18:05.950819 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "d8faedd5a03d40fcb6414a889e2dfb3e"
format_stamp: "Formatted at 2026-06-20 08:18:05 on dist-test-slave-65w8"
server_key: "0757812e8d2ec23ba02361d1fa2ca06e"
server_key_iv: "17ef343c2a49960e3b35e9c8739a1c1a"
server_key_version: "encryptionkey@0"
I20260620 08:18:05.952190 10184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:05.953138 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:05.953364 10184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:05.953516 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "d8faedd5a03d40fcb6414a889e2dfb3e"
format_stamp: "Formatted at 2026-06-20 08:18:05 on dist-test-slave-65w8"
server_key: "0757812e8d2ec23ba02361d1fa2ca06e"
server_key_iv: "17ef343c2a49960e3b35e9c8739a1c1a"
server_key_version: "encryptionkey@0"
I20260620 08:18:05.953595 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:05.978046 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:05.980496 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:05.980684 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:05.980950 10184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:05.981297 10184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:05.981355 10184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:05.981411 10184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:05.981439 10184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:05.987947 10184 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:38215
I20260620 08:18:05.988085 10312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:38215 every 8 connection(s)
I20260620 08:18:05.988382 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:18:05.990849 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10184
I20260620 08:18:05.991079 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:18:05.991295 25574 external_mini_cluster.cc:1428] Setting key 2d7dab04a704e8118a094bfbd0068a44
Jun 20 08:18:05 dist-test-slave-65w8 krb5kdc[10077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943485, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:05.999755 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:05.989227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:59841 (local address 127.24.249.190:42357)
0620 08:18:05.989292 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:05.989298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:05.989541 (+ 243us) server_negotiation.cc:407] Connection header received
0620 08:18:05.989999 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:05.990001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:05.990016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:05.990086 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:05.991123 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:05.991660 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:05.992874 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:05.993105 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:05.993199 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:05.995993 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:05.995995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:05.995995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:05.996004 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:05.997844 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:05.998541 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:05.998542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:05.998543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:05.998613 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:05.998889 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:05.998890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:05.998890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:05.999029 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:05.999245 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:05.999552 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 08:18:05.999655 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:18:05.999984 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:05.989134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42357 (local address 127.24.249.129:59841)
0620 08:18:05.989305 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:05.989348 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:05.989907 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:05.990141 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:05.990144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:05.990744 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:05.991006 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:05.991013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:05.991766 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:05.991767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:05.992732 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:05.992735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:05.993213 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:05.993214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:05.993357 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:05.994015 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:05.994023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:05.995861 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:05.998042 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:05.998051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:05.998053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:05.998311 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:05.998696 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:05.998697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:05.998697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:05.998818 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:05.999231 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:05.999240 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:05.999403 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 08:18:05.999528 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 08:18:05.999718 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:18:06.000517 10313 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42357
I20260620 08:18:06.000628 10313 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:06.000825 10313 heartbeater.cc:511] Master 127.24.249.190:42357 requested a full tablet report, sending...
I20260620 08:18:06.001372 10122 ts_manager.cc:194] Registered new tserver with Master: d8faedd5a03d40fcb6414a889e2dfb3e (127.24.249.129:38215)
I20260620 08:18:06.002188 10122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:59841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:18:06.022360 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42357
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:06.107187 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:06.107416 10318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:06.107443 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:06.108616 10318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:06.108657 10318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:06.108685 10318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:18:06.110154 10318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:06.110369 10318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:06.110562 10318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:06.116123 10323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.116123 10326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:06.116219 10318 server_base.cc:1034] running on GCE node
W20260620 08:18:06.116233 10324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:06.116660 10318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:06.116926 10318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:06.118116 10318 hybrid_clock.cc:648] HybridClock initialized: now 1781943486118083 us; error 52 us; skew 500 ppm
Jun 20 08:18:06 dist-test-slave-65w8 krb5kdc[10077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:06.120918 10318 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:18:06.121462 10318 webserver.cc:460] Webserver started at http://127.24.249.130:35493/ using document root <none> and password file <none>
I20260620 08:18:06.121719 10318 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:06.121790 10318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:06.122007 10318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:06.123152 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "ab2cbec3254d4230affa3eed14d4d0dd"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "12336d65807c7266070a8c44724f04b7"
server_key_iv: "b6293f4f99c3315994eff3e8fb728d88"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.123526 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "ab2cbec3254d4230affa3eed14d4d0dd"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "12336d65807c7266070a8c44724f04b7"
server_key_iv: "b6293f4f99c3315994eff3e8fb728d88"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.124949 10318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:18:06.125782 10333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.126014 10318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:06.126127 10318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "ab2cbec3254d4230affa3eed14d4d0dd"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "12336d65807c7266070a8c44724f04b7"
server_key_iv: "b6293f4f99c3315994eff3e8fb728d88"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.126204 10318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:06.151018 10318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:06.153206 10318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:06.153388 10318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:06.153674 10318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:06.154090 10318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:06.154145 10318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.154189 10318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:06.154217 10318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.160811 10318 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:46321
I20260620 08:18:06.160887 10446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:46321 every 8 connection(s)
I20260620 08:18:06.161252 10318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:18:06 dist-test-slave-65w8 krb5kdc[10077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:06.168187 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10318
I20260620 08:18:06.168322 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:18:06.168634 25574 external_mini_cluster.cc:1428] Setting key 3819474faa56584c2d20a66e58652e9d
I20260620 08:18:06.171994 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.162124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:56871 (local address 127.24.249.190:42357)
0620 08:18:06.162212 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.162215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.162386 (+ 171us) server_negotiation.cc:407] Connection header received
0620 08:18:06.162955 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.162957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.162970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.163050 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.164095 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.164607 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.165616 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.165830 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.166086 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.168570 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.168572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.168572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:06.168581 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.170356 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:06.170901 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:06.170903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:06.170903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:06.170962 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:06.171256 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:06.171257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:06.171257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:06.171409 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.171474 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.171843 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.171928 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:18:06.172278 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.162031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42357 (local address 127.24.249.130:56871)
0620 08:18:06.162252 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.162303 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.162838 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.163122 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.163127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.163788 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:06.163992 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.163996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.164705 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.164706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.165518 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.165520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.166096 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.166098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.166262 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.166802 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:06.166811 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:06.168457 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:06.170462 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:06.170471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:06.170473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:06.170804 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:06.171035 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:06.171036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:06.171036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:06.171191 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:06.171486 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:06.171493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:06.171676 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.171821 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.171999 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
I20260620 08:18:06.172905 10447 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42357
I20260620 08:18:06.173029 10447 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:06.173219 10447 heartbeater.cc:511] Master 127.24.249.190:42357 requested a full tablet report, sending...
I20260620 08:18:06.173610 10122 ts_manager.cc:194] Registered new tserver with Master: ab2cbec3254d4230affa3eed14d4d0dd (127.24.249.130:46321)
I20260620 08:18:06.174115 10122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:56871
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:18:06.196684 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42357
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:06.279145 10452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:06.279397 10452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:06.279439 10452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:06.280764 10452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:06.280845 10452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:06.280893 10452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:18:06.282680 10452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:06.283021 10452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:06.283278 10452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:06.288857 10457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.288844 10460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.288849 10458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:06.289125 10452 server_base.cc:1034] running on GCE node
I20260620 08:18:06.289326 10452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:06.289613 10452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:06.290791 10452 hybrid_clock.cc:648] HybridClock initialized: now 1781943486290749 us; error 60 us; skew 500 ppm
Jun 20 08:18:06 dist-test-slave-65w8 krb5kdc[10077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:06.293514 10452 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:18:06.294032 10452 webserver.cc:460] Webserver started at http://127.24.249.131:38791/ using document root <none> and password file <none>
I20260620 08:18:06.294283 10452 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:06.294364 10452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:06.294518 10452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:06.295596 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "8bc9ae8ae697493ba9dc97cb0287476c"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "1d8496e48bdb474892896562ec679f34"
server_key_iv: "bbc54b6ecdca3c4f3be018c34449d2fc"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.295964 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "8bc9ae8ae697493ba9dc97cb0287476c"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "1d8496e48bdb474892896562ec679f34"
server_key_iv: "bbc54b6ecdca3c4f3be018c34449d2fc"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.297412 10452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:06.298441 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.298661 10452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:18:06.298763 10452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "8bc9ae8ae697493ba9dc97cb0287476c"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "1d8496e48bdb474892896562ec679f34"
server_key_iv: "bbc54b6ecdca3c4f3be018c34449d2fc"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.298852 10452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:06.306504 10452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:06.308483 10452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:06.308642 10452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:06.308876 10452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:06.309188 10452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:06.309242 10452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.309283 10452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:06.309309 10452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.315907 10452 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:39645
I20260620 08:18:06.316066 10580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:39645 every 8 connection(s)
I20260620 08:18:06.316380 10452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:18:06.322048 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10452
I20260620 08:18:06.322163 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:18:06.322366 25574 external_mini_cluster.cc:1428] Setting key 37aebccea1f16d62b8a34f48c64db51e
Jun 20 08:18:06 dist-test-slave-65w8 krb5kdc[10077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943486, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:06.326226 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.317480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:57589 (local address 127.24.249.190:42357)
0620 08:18:06.317548 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.317550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.317560 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:06.318106 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.318107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.318118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.318184 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.319127 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.319644 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.320618 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.320805 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.320864 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.323316 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.323317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.323318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:06.323326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.324879 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:06.325258 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:06.325260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:06.325260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:06.325322 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:06.325585 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:06.325586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:06.325586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:06.325713 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.325780 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.326089 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.326161 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 08:18:06.326447 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.317146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42357 (local address 127.24.249.131:57589)
0620 08:18:06.317332 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.317368 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.317973 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.318241 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.318244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.318825 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:06.319030 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.319036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.319730 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.319731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.320537 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.320539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.320882 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.320883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.320996 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.321486 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:06.321494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:06.323189 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:06.324954 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:06.324962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:06.324963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:06.325173 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:06.325389 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:06.325390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:06.325390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:06.325500 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:06.325780 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:06.325785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:06.325952 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.326055 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.326179 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260620 08:18:06.327116 10581 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42357
I20260620 08:18:06.327242 10581 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:06.327492 10581 heartbeater.cc:511] Master 127.24.249.190:42357 requested a full tablet report, sending...
I20260620 08:18:06.327906 10122 ts_manager.cc:194] Registered new tserver with Master: 8bc9ae8ae697493ba9dc97cb0287476c (127.24.249.131:39645)
I20260620 08:18:06.328279 10122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:57589
I20260620 08:18:06.334543 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:18:06.342516 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.336059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.24.249.190:42357)
0620 08:18:06.336129 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.336131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.336227 (+ 96us) server_negotiation.cc:407] Connection header received
0620 08:18:06.336273 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.336274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.336284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.336337 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.336894 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.337374 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.338189 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.338370 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.338427 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.339626 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.339633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.339633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:06.339640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.341078 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:06.341487 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:06.341488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:06.341488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:06.341548 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:06.341807 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:06.341808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:06.341808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:06.342062 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.342152 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.342349 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.342448 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 08:18:06.344726 10122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:18:06.346397 10122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:18:06.353020 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.348748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38215 (local address 127.0.0.1:39214)
0620 08:18:06.348917 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.348958 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.349025 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.349334 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.349338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.349341 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:06.349421 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.349424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.350783 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.350784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.351887 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.351890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.352656 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.352657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.352739 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.352752 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.352857 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.352925 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":27,"spinlock_wait_cycles":384,"thread_start_us":102,"threads_started":1}
I20260620 08:18:06.353022 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.348951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.24.249.129:38215)
0620 08:18:06.349134 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.349139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.349153 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:18:06.349197 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.349199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.349217 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.349314 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:06.349518 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.350678 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.351980 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.352557 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.352648 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.352710 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.352878 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.352925 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260620 08:18:06.353403 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.348752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39645 (local address 127.0.0.1:39844)
0620 08:18:06.349135 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.349146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.349183 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.349454 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.349456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.349459 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:06.349505 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.349507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.350544 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.350545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.352043 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.352044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.353110 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.353111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.353181 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.353185 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.353248 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.353302 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":57,"threads_started":1}
I20260620 08:18:06.353631 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.348751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46321 (local address 127.0.0.1:45548)
0620 08:18:06.349385 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.349395 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.349444 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.349735 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.349736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.349738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:06.349780 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.349781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.351123 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.351124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.352278 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.352281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.353273 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.353274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.353324 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.353326 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.353372 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.353404 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":91,"threads_started":1}
I20260620 08:18:06.353668 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.348937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.24.249.130:46321)
0620 08:18:06.349338 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.349342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.349518 (+ 176us) server_negotiation.cc:407] Connection header received
0620 08:18:06.349568 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.349570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.349584 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.349654 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:06.349884 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.350906 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.352382 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.353120 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.353182 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.353239 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.353455 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.353590 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":85,"threads_started":1}
I20260620 08:18:06.353682 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.349044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.24.249.131:39645)
0620 08:18:06.349246 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.349250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.349259 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:06.349297 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.349299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.349314 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.349397 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:06.349611 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.350436 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.352126 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.352945 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.353020 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.353070 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.353555 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.353604 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260620 08:18:06.354063 10247 tablet_service.cc:1467] Processing CreateTablet for tablet d6e44d7135124a4fade272a35856a59e (DEFAULT_TABLE table=test-table [id=2894097166194cf8b18cabd432633c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:06.354461 10247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e44d7135124a4fade272a35856a59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:06.354388 10515 tablet_service.cc:1467] Processing CreateTablet for tablet d6e44d7135124a4fade272a35856a59e (DEFAULT_TABLE table=test-table [id=2894097166194cf8b18cabd432633c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:06.354460 10381 tablet_service.cc:1467] Processing CreateTablet for tablet d6e44d7135124a4fade272a35856a59e (DEFAULT_TABLE table=test-table [id=2894097166194cf8b18cabd432633c8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:06.354715 10515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e44d7135124a4fade272a35856a59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:06.354768 10381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e44d7135124a4fade272a35856a59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:06.357653 10598 tablet_bootstrap.cc:492] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Bootstrap starting.
I20260620 08:18:06.357896 10599 tablet_bootstrap.cc:492] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd: Bootstrap starting.
I20260620 08:18:06.358455 10598 tablet_bootstrap.cc:654] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:06.358526 10599 tablet_bootstrap.cc:654] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:06.359125 10598 log.cc:824] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:06.359125 10599 log.cc:824] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:06.359464 10600 tablet_bootstrap.cc:492] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e: Bootstrap starting.
I20260620 08:18:06.359848 10599 tablet_bootstrap.cc:492] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd: No bootstrap required, opened a new log
I20260620 08:18:06.359915 10599 ts_tablet_manager.cc:1397] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:18:06.360054 10600 tablet_bootstrap.cc:654] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:06.360535 10600 log.cc:824] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:06.361244 10600 tablet_bootstrap.cc:492] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e: No bootstrap required, opened a new log
I20260620 08:18:06.361328 10600 ts_tablet_manager.cc:1397] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:06.361737 10599 raft_consensus.cc:348] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.361955 10599 raft_consensus.cc:374] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:06.362006 10599 raft_consensus.cc:729] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab2cbec3254d4230affa3eed14d4d0dd, State: Initialized, Role: FOLLOWER
I20260620 08:18:06.362139 10599 consensus_queue.cc:260] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.362435 10599 ts_tablet_manager.cc:1428] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 08:18:06.362542 10598 tablet_bootstrap.cc:492] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: No bootstrap required, opened a new log
I20260620 08:18:06.362587 10447 heartbeater.cc:503] Master 127.24.249.190:42357 was elected leader, sending a full tablet report...
I20260620 08:18:06.362622 10598 ts_tablet_manager.cc:1397] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 08:18:06.363371 10600 raft_consensus.cc:348] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.363549 10600 raft_consensus.cc:374] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:06.363587 10600 raft_consensus.cc:729] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8faedd5a03d40fcb6414a889e2dfb3e, State: Initialized, Role: FOLLOWER
I20260620 08:18:06.363693 10600 consensus_queue.cc:260] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.363961 10600 ts_tablet_manager.cc:1428] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:06.364017 10313 heartbeater.cc:503] Master 127.24.249.190:42357 was elected leader, sending a full tablet report...
I20260620 08:18:06.364727 10598 raft_consensus.cc:348] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.364904 10598 raft_consensus.cc:374] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:06.364959 10598 raft_consensus.cc:729] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc9ae8ae697493ba9dc97cb0287476c, State: Initialized, Role: FOLLOWER
I20260620 08:18:06.365075 10598 consensus_queue.cc:260] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.365350 10598 ts_tablet_manager.cc:1428] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:06.365413 10581 heartbeater.cc:503] Master 127.24.249.190:42357 was elected leader, sending a full tablet report...
I20260620 08:18:06.368496 10604 raft_consensus.cc:482] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:18:06.368620 10604 raft_consensus.cc:504] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.369006 10604 leader_election.cc:288] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8faedd5a03d40fcb6414a889e2dfb3e (127.24.249.129:38215), 8bc9ae8ae697493ba9dc97cb0287476c (127.24.249.131:39645)
I20260620 08:18:06.372115 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.369227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:47321 (local address 127.24.249.129:38215)
0620 08:18:06.369274 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.369276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.369474 (+ 198us) server_negotiation.cc:407] Connection header received
0620 08:18:06.369532 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.369533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.369543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.369599 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:06.369754 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.370303 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.371345 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.371736 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.371842 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.371851 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.372006 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.372042 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:18:06.372119 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.369394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:50155 (local address 127.24.249.131:39645)
0620 08:18:06.369452 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.369454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.369462 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:06.369486 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.369487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.369495 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.369589 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:06.369752 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.370310 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.371361 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.371744 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.371851 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.371860 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.372017 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.372055 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:18:06.372130 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.369168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39645 (local address 127.24.249.130:50155)
0620 08:18:06.369207 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.369220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.369263 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.369599 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.369600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.369602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:06.369673 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.369674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.370400 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.370401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.371242 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.371244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.371852 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.371853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.371942 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.371945 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.372012 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.372053 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":30}
I20260620 08:18:06.372133 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.369178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38215 (local address 127.24.249.130:47321)
0620 08:18:06.369410 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.369417 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.369480 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.369604 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.369605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.369607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:06.369679 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.369680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.370388 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.370389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.371238 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.371240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.371836 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.371837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.371911 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.371915 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.371998 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.372042 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":50,"thread_start_us":145,"threads_started":1}
I20260620 08:18:06.372454 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e44d7135124a4fade272a35856a59e" candidate_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" is_pre_election: true
I20260620 08:18:06.372464 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e44d7135124a4fade272a35856a59e" candidate_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" is_pre_election: true
I20260620 08:18:06.372599 10267 raft_consensus.cc:2454] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab2cbec3254d4230affa3eed14d4d0dd in term 0.
I20260620 08:18:06.372598 10535 raft_consensus.cc:2454] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab2cbec3254d4230affa3eed14d4d0dd in term 0.
I20260620 08:18:06.372813 10336 leader_election.cc:302] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab2cbec3254d4230affa3eed14d4d0dd, d8faedd5a03d40fcb6414a889e2dfb3e; no voters:
I20260620 08:18:06.372995 10604 raft_consensus.cc:2781] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:18:06.373052 10604 raft_consensus.cc:482] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:18:06.373078 10604 raft_consensus.cc:3037] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:06.373708 10604 raft_consensus.cc:504] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.373903 10604 leader_election.cc:288] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [CANDIDATE]: Term 1 election: Requested vote from peers d8faedd5a03d40fcb6414a889e2dfb3e (127.24.249.129:38215), 8bc9ae8ae697493ba9dc97cb0287476c (127.24.249.131:39645)
I20260620 08:18:06.374042 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e44d7135124a4fade272a35856a59e" candidate_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e"
I20260620 08:18:06.374059 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e44d7135124a4fade272a35856a59e" candidate_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc9ae8ae697493ba9dc97cb0287476c"
I20260620 08:18:06.374122 10535 raft_consensus.cc:3037] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:06.374125 10267 raft_consensus.cc:3037] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:06.374998 10535 raft_consensus.cc:2454] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab2cbec3254d4230affa3eed14d4d0dd in term 1.
I20260620 08:18:06.374998 10267 raft_consensus.cc:2454] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab2cbec3254d4230affa3eed14d4d0dd in term 1.
I20260620 08:18:06.375160 10335 leader_election.cc:302] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bc9ae8ae697493ba9dc97cb0287476c, ab2cbec3254d4230affa3eed14d4d0dd; no voters:
I20260620 08:18:06.375269 10604 raft_consensus.cc:2781] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:06.375403 10604 raft_consensus.cc:686] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [term 1 LEADER]: Becoming Leader. State: Replica: ab2cbec3254d4230affa3eed14d4d0dd, State: Running, Role: LEADER
I20260620 08:18:06.375501 10604 consensus_queue.cc:237] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } }
I20260620 08:18:06.376400 10120 catalog_manager.cc:5495] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd reported cstate change: term changed from 0 to 1, leader changed from <none> to ab2cbec3254d4230affa3eed14d4d0dd (127.24.249.130). New cstate: current_term: 1 leader_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab2cbec3254d4230affa3eed14d4d0dd" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20260620 08:18:06.385639 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.382969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.24.249.130:46321)
0620 08:18:06.383034 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.383036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.383046 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:06.383092 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.383092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.383109 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.383192 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:06.383384 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.383859 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.384790 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.384987 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.385064 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.385246 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:06.385345 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:06.385474 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.385501 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.385521 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 08:18:06.387723 10535 raft_consensus.cc:1261] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 1 FOLLOWER]: Refusing update from remote peer ab2cbec3254d4230affa3eed14d4d0dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:06.387754 10267 raft_consensus.cc:1261] T d6e44d7135124a4fade272a35856a59e P d8faedd5a03d40fcb6414a889e2dfb3e [term 1 FOLLOWER]: Refusing update from remote peer ab2cbec3254d4230affa3eed14d4d0dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:06.388154 10604 consensus_queue.cc:1035] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc9ae8ae697493ba9dc97cb0287476c" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:06.388345 10608 consensus_queue.cc:1035] T d6e44d7135124a4fade272a35856a59e P ab2cbec3254d4230affa3eed14d4d0dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8faedd5a03d40fcb6414a889e2dfb3e" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 38215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:06.393308 10120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:06.393438 10120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:06.394296 10120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bdaa8bb85614e33a137ae21308a72d2: Sending DeleteTablet for 3 replicas of tablet d6e44d7135124a4fade272a35856a59e
I20260620 08:18:06.394663 10247 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e44d7135124a4fade272a35856a59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:06 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:39214
I20260620 08:18:06.394757 10515 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e44d7135124a4fade272a35856a59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:06 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:39844
I20260620 08:18:06.395030 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10184
I20260620 08:18:06.395620 10381 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e44d7135124a4fade272a35856a59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:06 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:45548
I20260620 08:18:06.396011 10622 tablet_replica.cc:330] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: stopping tablet replica
I20260620 08:18:06.396159 10622 raft_consensus.cc:2229] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:06.396287 10622 raft_consensus.cc:2258] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:06.396785 10622 ts_tablet_manager.cc:1905] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:06.398669 10622 ts_tablet_manager.cc:1918] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:06.398785 10622 log.cc:1196] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/d6e44d7135124a4fade272a35856a59e
I20260620 08:18:06.399183 10622 ts_tablet_manager.cc:1939] T d6e44d7135124a4fade272a35856a59e P 8bc9ae8ae697493ba9dc97cb0287476c: Deleting consensus metadata
I20260620 08:18:06.399689 10107 catalog_manager.cc:4841] TS 8bc9ae8ae697493ba9dc97cb0287476c (127.24.249.131:39645): tablet d6e44d7135124a4fade272a35856a59e (table test-table [id=2894097166194cf8b18cabd432633c8a]) successfully deleted
W20260620 08:18:06.402344 10108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:38215 (error 108)
I20260620 08:18:06.402952 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10318
I20260620 08:18:06.410684 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.405806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:38215 (local address 127.0.0.1:39226)
0620 08:18:06.410573 (+ 4767us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.410636 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4737}
I20260620 08:18:06.410792 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.410670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46321 (local address 127.0.0.1:45560)
0620 08:18:06.410724 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.410752 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.130:46321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 08:18:06.410882 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10452
W20260620 08:18:06.410981 10109 catalog_manager.cc:4584] TS ab2cbec3254d4230affa3eed14d4d0dd (127.24.249.130:46321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6e44d7135124a4fade272a35856a59e: Network error: Client connection negotiation failed: client connection to 127.24.249.130:46321: connect: Connection refused (error 111)
I20260620 08:18:06.416314 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10090
2026-06-20T08:18:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T08:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:06Z Disabled control of system clock
I20260620 08:18:06.447407 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34503
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:06.530066 10628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:06.530298 10628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:06.530319 10628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:06.531560 10628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:06.531613 10628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:06.531626 10628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:06.531639 10628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:06.531652 10628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:06.533006 10628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34503
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:06.533277 10628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:06.533572 10628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:06.538447 10633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.538447 10634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.538479 10636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:06.538733 10628 server_base.cc:1034] running on GCE node
I20260620 08:18:06.539004 10628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:06.539300 10628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:06.540457 10628 hybrid_clock.cc:648] HybridClock initialized: now 1781943486540433 us; error 46 us; skew 500 ppm
I20260620 08:18:06.541803 10628 webserver.cc:460] Webserver started at http://127.24.249.190:44859/ using document root <none> and password file <none>
I20260620 08:18:06.542114 10628 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:06.542201 10628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:06.542366 10628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:06.543570 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "e9698da555d44a9cad33f839d3b2995d"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "93dc88316d7d2e438131067f5bab1e0a"
server_key_iv: "e1dbf0179f00ee044d693ba556126416"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.543972 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "e9698da555d44a9cad33f839d3b2995d"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "93dc88316d7d2e438131067f5bab1e0a"
server_key_iv: "e1dbf0179f00ee044d693ba556126416"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.545434 10628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:06.546362 10642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.546553 10628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:18:06.546666 10628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "e9698da555d44a9cad33f839d3b2995d"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "93dc88316d7d2e438131067f5bab1e0a"
server_key_iv: "e1dbf0179f00ee044d693ba556126416"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.546758 10628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:06.556180 10628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:06.556484 10628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:06.556632 10628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:06.560616 10628 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:34503
I20260620 08:18:06.560732 10694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:34503 every 8 connection(s)
I20260620 08:18:06.561033 10628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:06.562134 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10628
I20260620 08:18:06.562253 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:06.562243 10695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:06.562520 25574 external_mini_cluster.cc:1428] Setting key b9f6a21b47570469ab1b2c5571813420
I20260620 08:18:06.565343 10695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d: Bootstrap starting.
I20260620 08:18:06.566032 10695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:06.566371 10695 log.cc:824] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:06.567195 10695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d: No bootstrap required, opened a new log
I20260620 08:18:06.568517 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.563113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.24.249.190:34503)
0620 08:18:06.563368 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.563375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.563400 (+ 25us) server_negotiation.cc:407] Connection header received
0620 08:18:06.563875 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.563881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.563948 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.564096 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.564326 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.565199 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.566156 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.566812 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.566957 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.567735 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.567740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.567740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:06.567955 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.568003 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:06.568017 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.568096 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.568175 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.568264 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260620 08:18:06.569765 10695 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9698da555d44a9cad33f839d3b2995d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34503 } }
I20260620 08:18:06.569975 10695 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:06.570005 10695 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9698da555d44a9cad33f839d3b2995d, State: Initialized, Role: FOLLOWER
I20260620 08:18:06.570128 10695 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9698da555d44a9cad33f839d3b2995d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34503 } }
I20260620 08:18:06.570216 10695 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:06.570242 10695 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:06.570308 10695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:06.570928 10695 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9698da555d44a9cad33f839d3b2995d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34503 } }
I20260620 08:18:06.571028 10695 leader_election.cc:302] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9698da555d44a9cad33f839d3b2995d; no voters:
I20260620 08:18:06.571209 10695 leader_election.cc:288] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:06.571348 10700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:06.571493 10695 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:06.571575 10700 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [term 1 LEADER]: Becoming Leader. State: Replica: e9698da555d44a9cad33f839d3b2995d, State: Running, Role: LEADER
I20260620 08:18:06.571672 10700 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9698da555d44a9cad33f839d3b2995d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34503 } }
I20260620 08:18:06.572075 10702 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9698da555d44a9cad33f839d3b2995d. Latest consensus state: current_term: 1 leader_uuid: "e9698da555d44a9cad33f839d3b2995d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9698da555d44a9cad33f839d3b2995d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34503 } } }
I20260620 08:18:06.572180 10702 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:06.572072 10701 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9698da555d44a9cad33f839d3b2995d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9698da555d44a9cad33f839d3b2995d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34503 } } }
I20260620 08:18:06.572407 10709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:06.572973 10709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:06.572420 10701 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:06.574728 10709 catalog_manager.cc:1347] Generated new cluster ID: 8ff5f82160ec447195a629a6773ca763
I20260620 08:18:06.574795 10709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:06.581425 10709 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:06.582197 10709 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:06.585031 10709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d: Generated new TSK 0
I20260620 08:18:06.585210 10709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:06.587517 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34503
--builtin_ntp_servers=127.24.249.148:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:06.673342 10719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:06.673614 10719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:06.673659 10719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:06.675058 10719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:06.675132 10719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:06.675182 10719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:18:06.676729 10719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:06.677021 10719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:06.677357 10719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:06.683446 10727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.683509 10725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.683452 10724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:06.684214 10719 server_base.cc:1034] running on GCE node
I20260620 08:18:06.684458 10719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:06.684712 10719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:06.685950 10719 hybrid_clock.cc:648] HybridClock initialized: now 1781943486685924 us; error 38 us; skew 500 ppm
I20260620 08:18:06.687430 10719 webserver.cc:460] Webserver started at http://127.24.249.129:34173/ using document root <none> and password file <none>
I20260620 08:18:06.687696 10719 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:06.687770 10719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:06.688014 10719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:06.689177 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "4b12f0d344214fe9acdac6fd88f6c2ee"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "f0fca6e7cebc8ffc191c7b3d2712bf52"
server_key_iv: "b3db84201ba9a806b1a634cf5c3087c4"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.689548 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "4b12f0d344214fe9acdac6fd88f6c2ee"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "f0fca6e7cebc8ffc191c7b3d2712bf52"
server_key_iv: "b3db84201ba9a806b1a634cf5c3087c4"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.690959 10719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:06.691766 10733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.691988 10719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:06.692090 10719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "4b12f0d344214fe9acdac6fd88f6c2ee"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "f0fca6e7cebc8ffc191c7b3d2712bf52"
server_key_iv: "b3db84201ba9a806b1a634cf5c3087c4"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.692179 10719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:06.703744 10719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:06.704120 10719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:06.704291 10719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:06.704560 10719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:06.704941 10719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:06.705026 10719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.705113 10719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:06.705148 10719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.712175 10719 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:43537
I20260620 08:18:06.712224 10846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:43537 every 8 connection(s)
I20260620 08:18:06.712601 10719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:18:06.713076 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10719
I20260620 08:18:06.713183 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:18:06.713419 25574 external_mini_cluster.cc:1428] Setting key dad68ccde496a5d6333651170d389578
I20260620 08:18:06.714610 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34503
--builtin_ntp_servers=127.24.249.148:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:18:06.719156 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.713426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:33739 (local address 127.24.249.190:34503)
0620 08:18:06.713481 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.713483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.713823 (+ 340us) server_negotiation.cc:407] Connection header received
0620 08:18:06.714542 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.714543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.714561 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.714610 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.715425 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.715977 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.717204 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.717587 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.717676 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.718736 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.718738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.718738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:06.718744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.718764 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:06.718769 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.718838 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.718994 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.719030 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:06.719282 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.713446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34503 (local address 127.24.249.129:33739)
0620 08:18:06.713668 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.713717 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.714409 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.714696 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.714701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.714713 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:06.715324 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.715330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.716142 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.716144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.717092 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.717094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.717699 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.717700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.717821 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.718534 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:06.718534 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:06.718580 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:06.718614 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:06.718874 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:06.718884 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:06.718888 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.718969 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.719062 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260620 08:18:06.719811 10847 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34503
I20260620 08:18:06.719923 10847 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:06.720139 10847 heartbeater.cc:511] Master 127.24.249.190:34503 requested a full tablet report, sending...
I20260620 08:18:06.720690 10659 ts_manager.cc:194] Registered new tserver with Master: 4b12f0d344214fe9acdac6fd88f6c2ee (127.24.249.129:43537)
I20260620 08:18:06.721321 10659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.129:33739
W20260620 08:18:06.802353 10850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:06.802589 10850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:06.802614 10850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:06.803877 10850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:06.803926 10850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:06.803952 10850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:18:06.805713 10850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:06.806137 10850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:06.807003 10850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:06.814136 10855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.814136 10858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.814162 10856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:06.814529 10850 server_base.cc:1034] running on GCE node
I20260620 08:18:06.814821 10850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:06.815132 10850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:06.816332 10850 hybrid_clock.cc:648] HybridClock initialized: now 1781943486816298 us; error 55 us; skew 500 ppm
I20260620 08:18:06.818059 10850 webserver.cc:460] Webserver started at http://127.24.249.130:38889/ using document root <none> and password file <none>
I20260620 08:18:06.818344 10850 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:06.818423 10850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:06.818583 10850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:06.820009 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "67bbfd2cc53d431a922a49f6e5ed4d58"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "6d705ccc510d08e6c83d61d7c8c5291e"
server_key_iv: "77fc6deef4e97a01c6c7bd4dc0a45a7b"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.820580 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "67bbfd2cc53d431a922a49f6e5ed4d58"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "6d705ccc510d08e6c83d61d7c8c5291e"
server_key_iv: "77fc6deef4e97a01c6c7bd4dc0a45a7b"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.822237 10850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 08:18:06.823179 10864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.823519 10850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:06.823742 10850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "67bbfd2cc53d431a922a49f6e5ed4d58"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "6d705ccc510d08e6c83d61d7c8c5291e"
server_key_iv: "77fc6deef4e97a01c6c7bd4dc0a45a7b"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.823843 10850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:06.841421 10850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:06.841831 10850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:06.842106 10850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:06.842533 10850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:06.842968 10850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:06.843039 10850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.843084 10850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:06.843137 10850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.849606 10850 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:46353
I20260620 08:18:06.849680 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:46353 every 8 connection(s)
I20260620 08:18:06.850117 10850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:18:06.851111 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10850
I20260620 08:18:06.851213 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:18:06.851429 25574 external_mini_cluster.cc:1428] Setting key 475a76e67b2722cce2174bfde2ef0334
I20260620 08:18:06.852697 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34503
--builtin_ntp_servers=127.24.249.148:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:18:06.856618 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.850897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:52117 (local address 127.24.249.190:34503)
0620 08:18:06.850952 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.850954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.851294 (+ 340us) server_negotiation.cc:407] Connection header received
0620 08:18:06.852000 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.852001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.852011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.852095 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.853107 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.853658 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.854967 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.855185 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.855265 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.856265 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.856267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.856267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:06.856273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.856289 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:06.856296 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.856367 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.856506 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.856535 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:18:06.856858 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.850895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34503 (local address 127.24.249.130:52117)
0620 08:18:06.851147 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.851202 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.851870 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.852159 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.852162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.852171 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:06.853005 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.853013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.853762 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.853763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.854842 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.854845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.855287 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.855288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.855440 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.856085 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:06.856085 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:06.856114 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:06.856137 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:06.856367 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:06.856413 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:06.856416 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.856508 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.856617 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":117,"threads_started":1}
I20260620 08:18:06.857378 10978 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34503
I20260620 08:18:06.857460 10978 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:06.857666 10978 heartbeater.cc:511] Master 127.24.249.190:34503 requested a full tablet report, sending...
I20260620 08:18:06.858111 10659 ts_manager.cc:194] Registered new tserver with Master: 67bbfd2cc53d431a922a49f6e5ed4d58 (127.24.249.130:46353)
I20260620 08:18:06.858657 10659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.130:52117
W20260620 08:18:06.938971 10981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:06.939180 10981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:06.939203 10981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:06.940428 10981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:06.940472 10981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:06.940500 10981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:18:06.941817 10981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:34503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:06.942037 10981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:06.942288 10981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:18:06.947824 10981 server_base.cc:1034] running on GCE node
W20260620 08:18:06.947839 10989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.947793 10986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:06.948014 10987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:06.948266 10981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:06.948557 10981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:06.949730 10981 hybrid_clock.cc:648] HybridClock initialized: now 1781943486949714 us; error 31 us; skew 500 ppm
I20260620 08:18:06.951508 10981 webserver.cc:460] Webserver started at http://127.24.249.131:45213/ using document root <none> and password file <none>
I20260620 08:18:06.951807 10981 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:06.951889 10981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:06.952059 10981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:06.953244 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "200ba047bfaf481f8953be562ff2c096"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "3b71760ff4d7c57578216b2487b2f6c1"
server_key_iv: "1e714c6b7740cb581a63a01d0a392f99"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.953657 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "200ba047bfaf481f8953be562ff2c096"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "3b71760ff4d7c57578216b2487b2f6c1"
server_key_iv: "1e714c6b7740cb581a63a01d0a392f99"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.955272 10981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:06.956180 10995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.956399 10981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:06.956544 10981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "200ba047bfaf481f8953be562ff2c096"
format_stamp: "Formatted at 2026-06-20 08:18:06 on dist-test-slave-65w8"
server_key: "3b71760ff4d7c57578216b2487b2f6c1"
server_key_iv: "1e714c6b7740cb581a63a01d0a392f99"
server_key_version: "encryptionkey@0"
I20260620 08:18:06.956629 10981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:06.973594 10981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:06.974045 10981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:06.974229 10981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:06.974514 10981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:06.974905 10981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:06.974965 10981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.975006 10981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:06.975037 10981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:06.981789 10981 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:45747
I20260620 08:18:06.981833 11108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:45747 every 8 connection(s)
I20260620 08:18:06.982286 10981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:18:06.987761 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.983077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:33941 (local address 127.24.249.190:34503)
0620 08:18:06.983160 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.983162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.983402 (+ 240us) server_negotiation.cc:407] Connection header received
0620 08:18:06.983893 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.983894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.983905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.983990 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.984654 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.985246 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.986333 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.986541 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.986607 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.987524 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.987525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.987525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:06.987530 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.987545 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:06.987549 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.987607 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.987678 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.987700 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 08:18:06.987970 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.983053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34503 (local address 127.24.249.131:33941)
0620 08:18:06.983277 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:06.983313 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:06.983800 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:06.984046 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:06.984049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:06.984057 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:06.984555 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.984559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.985349 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.985350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.986235 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:06.986238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.986625 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.986626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:06.986736 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.987353 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:06.987354 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:06.987382 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:06.987405 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:06.987610 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:06.987613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:06.987616 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:06.987663 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 08:18:06.987743 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260620 08:18:06.988569 11109 heartbeater.cc:347] Connected to a master server at 127.24.249.190:34503
I20260620 08:18:06.988644 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 10981
I20260620 08:18:06.988682 11109 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:06.988740 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:18:06.988920 11109 heartbeater.cc:511] Master 127.24.249.190:34503 requested a full tablet report, sending...
I20260620 08:18:06.988955 25574 external_mini_cluster.cc:1428] Setting key 115b5c25defdef5f520b410ead98dceb
I20260620 08:18:06.989362 10659 ts_manager.cc:194] Registered new tserver with Master: 200ba047bfaf481f8953be562ff2c096 (127.24.249.131:45747)
I20260620 08:18:06.989914 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:18:06.989910 10659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.131:33941
I20260620 08:18:06.994627 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:06.991506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.24.249.190:34503)
0620 08:18:06.991591 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:06.991593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:06.991711 (+ 118us) server_negotiation.cc:407] Connection header received
0620 08:18:06.991777 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:06.991778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:06.991789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:06.991871 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:06.992041 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.992521 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.993291 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:06.993491 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:06.993568 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:06.994369 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:06.994371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:06.994371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:06.994376 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:06.994390 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:06.994394 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:06.994455 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:06.994519 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 08:18:06.994540 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:18:06.996989 10659 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:18:06.998700 10659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:18:07.006913 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.002265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.24.249.131:45747)
0620 08:18:07.002725 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:07.002730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:07.002741 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:18:07.002790 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:07.002792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:07.002809 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:07.002932 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:07.003134 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.004140 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.005775 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.006500 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.006591 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.006686 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:07.006781 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 08:18:07.006819 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":88,"threads_started":1}
I20260620 08:18:07.006902 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.002001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:45747 (local address 127.0.0.1:35936)
0620 08:18:07.002176 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:07.002218 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:07.002318 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:07.002959 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:07.002959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:07.002961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:07.003055 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.003057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.004245 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.004246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.005675 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.005678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.006588 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.006589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.006670 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.006683 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 08:18:07.006758 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 08:18:07.006818 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":15,"thread_start_us":100,"threads_started":1}
I20260620 08:18:07.007457 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.002131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:43537 (local address 127.0.0.1:43796)
0620 08:18:07.002652 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:07.002665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:07.002728 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:07.003452 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:07.003453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:07.003455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:07.003502 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.003504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.004859 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.004860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.006048 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.006050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.007184 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.007185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.007258 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.007263 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:18:07.007318 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:18:07.007365 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":99,"threads_started":1}
I20260620 08:18:07.007593 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.002001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46353 (local address 127.0.0.1:36154)
0620 08:18:07.002176 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:07.002218 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:07.002286 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:07.002949 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:07.002953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:07.002956 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:07.003055 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.003058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.005279 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.005280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.006216 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.006219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.007274 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.007275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.007342 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.007344 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:18:07.007408 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:18:07.007442 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":5,"thread_start_us":57,"threads_started":1}
I20260620 08:18:07.007755 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.002451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.24.249.130:46353)
0620 08:18:07.002675 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:07.002681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:07.002694 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:18:07.002736 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:07.002738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:07.002755 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:07.002835 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:07.004250 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.005193 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.006317 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.007088 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.007159 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.007239 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:07.007504 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 08:18:07.007555 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260620 08:18:07.007823 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.002439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.24.249.129:43537)
0620 08:18:07.003206 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:07.003209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:07.003217 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:07.003300 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:07.003302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:07.003314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:07.003377 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:07.003619 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.004754 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.006140 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.006966 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.007049 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.007143 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:07.007665 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 08:18:07.007724 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":103,"threads_started":1}
I20260620 08:18:07.007988 11043 tablet_service.cc:1467] Processing CreateTablet for tablet 13811a326640478abed9603e32db378b (DEFAULT_TABLE table=test-table [id=b6c5114f52bd457b9ccd51516df1a858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:07.008410 11043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13811a326640478abed9603e32db378b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:07.008421 10912 tablet_service.cc:1467] Processing CreateTablet for tablet 13811a326640478abed9603e32db378b (DEFAULT_TABLE table=test-table [id=b6c5114f52bd457b9ccd51516df1a858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:07.008509 10781 tablet_service.cc:1467] Processing CreateTablet for tablet 13811a326640478abed9603e32db378b (DEFAULT_TABLE table=test-table [id=b6c5114f52bd457b9ccd51516df1a858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:07.008685 10912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13811a326640478abed9603e32db378b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:07.008812 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13811a326640478abed9603e32db378b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:07.012095 11123 tablet_bootstrap.cc:492] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Bootstrap starting.
I20260620 08:18:07.012645 11124 tablet_bootstrap.cc:492] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Bootstrap starting.
I20260620 08:18:07.013263 11123 tablet_bootstrap.cc:654] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:07.013263 11124 tablet_bootstrap.cc:654] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:07.013624 11123 log.cc:824] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:07.013643 11124 log.cc:824] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:07.014012 11125 tablet_bootstrap.cc:492] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee: Bootstrap starting.
I20260620 08:18:07.014472 11124 tablet_bootstrap.cc:492] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: No bootstrap required, opened a new log
I20260620 08:18:07.014487 11123 tablet_bootstrap.cc:492] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: No bootstrap required, opened a new log
I20260620 08:18:07.014555 11123 ts_tablet_manager.cc:1397] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:18:07.014555 11124 ts_tablet_manager.cc:1397] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:07.014644 11125 tablet_bootstrap.cc:654] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:07.014925 11125 log.cc:824] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:07.015630 11125 tablet_bootstrap.cc:492] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee: No bootstrap required, opened a new log
I20260620 08:18:07.015718 11125 ts_tablet_manager.cc:1397] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:07.017315 11124 raft_consensus.cc:348] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.017282 11123 raft_consensus.cc:348] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.017479 11123 raft_consensus.cc:374] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:07.017479 11124 raft_consensus.cc:374] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:07.017516 11124 raft_consensus.cc:729] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200ba047bfaf481f8953be562ff2c096, State: Initialized, Role: FOLLOWER
I20260620 08:18:07.017516 11123 raft_consensus.cc:729] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bbfd2cc53d431a922a49f6e5ed4d58, State: Initialized, Role: FOLLOWER
I20260620 08:18:07.017663 11123 consensus_queue.cc:260] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.017663 11124 consensus_queue.cc:260] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.017966 11123 ts_tablet_manager.cc:1428] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 08:18:07.017966 11124 ts_tablet_manager.cc:1428] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 08:18:07.017969 11125 raft_consensus.cc:348] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.018103 10978 heartbeater.cc:503] Master 127.24.249.190:34503 was elected leader, sending a full tablet report...
I20260620 08:18:07.018144 11125 raft_consensus.cc:374] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:07.018189 11125 raft_consensus.cc:729] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b12f0d344214fe9acdac6fd88f6c2ee, State: Initialized, Role: FOLLOWER
I20260620 08:18:07.018296 11125 consensus_queue.cc:260] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.018460 11109 heartbeater.cc:503] Master 127.24.249.190:34503 was elected leader, sending a full tablet report...
I20260620 08:18:07.018601 10847 heartbeater.cc:503] Master 127.24.249.190:34503 was elected leader, sending a full tablet report...
I20260620 08:18:07.018585 11125 ts_tablet_manager.cc:1428] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:07.090123 11130 raft_consensus.cc:482] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:18:07.090318 11130 raft_consensus.cc:504] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.090660 11130 leader_election.cc:288] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67bbfd2cc53d431a922a49f6e5ed4d58 (127.24.249.130:46353), 4b12f0d344214fe9acdac6fd88f6c2ee (127.24.249.129:43537)
I20260620 08:18:07.093999 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.090924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46353 (local address 127.24.249.131:37437)
0620 08:18:07.091099 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:07.091109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:07.091156 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:07.091376 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:07.091378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:07.091380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:07.091461 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.091462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.092133 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.092134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.092989 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.092992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.093673 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.093674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.093749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.093753 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:07.093814 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:18:07.093848 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":24,"thread_start_us":99,"threads_started":1}
I20260620 08:18:07.094069 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.091053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:37437 (local address 127.24.249.130:46353)
0620 08:18:07.091112 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:07.091114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:07.091146 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:18:07.091217 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:07.091218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:07.091226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:07.091282 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:07.091547 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.092035 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.093107 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.093492 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.093554 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.093561 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:07.093887 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 08:18:07.093953 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:18:07.094089 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.091088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:34183 (local address 127.24.249.129:43537)
0620 08:18:07.091140 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:07.091146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:07.091155 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:07.091182 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:07.091183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:07.091192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:07.091328 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:07.091585 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.092045 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.093140 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.093512 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.093707 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.093715 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:07.093958 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 08:18:07.094007 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:18:07.094143 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.090894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:43537 (local address 127.24.249.131:34183)
0620 08:18:07.090954 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:07.090966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:07.091014 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:07.091382 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:07.091383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:07.091384 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:07.091470 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.091472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.092149 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.092150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.092993 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:07.092996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.093725 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.093726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:07.093835 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.093840 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:18:07.093948 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 08:18:07.093994 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":5}
I20260620 08:18:07.094666 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13811a326640478abed9603e32db378b" candidate_uuid: "200ba047bfaf481f8953be562ff2c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" is_pre_election: true
I20260620 08:18:07.094652 10932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13811a326640478abed9603e32db378b" candidate_uuid: "200ba047bfaf481f8953be562ff2c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" is_pre_election: true
I20260620 08:18:07.094828 10932 raft_consensus.cc:2454] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 200ba047bfaf481f8953be562ff2c096 in term 0.
I20260620 08:18:07.094828 10801 raft_consensus.cc:2454] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 200ba047bfaf481f8953be562ff2c096 in term 0.
I20260620 08:18:07.095088 10998 leader_election.cc:302] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200ba047bfaf481f8953be562ff2c096, 4b12f0d344214fe9acdac6fd88f6c2ee; no voters:
I20260620 08:18:07.095278 11130 raft_consensus.cc:2781] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:18:07.095335 11130 raft_consensus.cc:482] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:18:07.095358 11130 raft_consensus.cc:3037] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:07.096266 11130 raft_consensus.cc:504] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.096434 11130 leader_election.cc:288] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [CANDIDATE]: Term 1 election: Requested vote from peers 67bbfd2cc53d431a922a49f6e5ed4d58 (127.24.249.130:46353), 4b12f0d344214fe9acdac6fd88f6c2ee (127.24.249.129:43537)
I20260620 08:18:07.096596 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13811a326640478abed9603e32db378b" candidate_uuid: "200ba047bfaf481f8953be562ff2c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee"
I20260620 08:18:07.096599 10932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13811a326640478abed9603e32db378b" candidate_uuid: "200ba047bfaf481f8953be562ff2c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58"
I20260620 08:18:07.096684 10801 raft_consensus.cc:3037] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:07.096712 10932 raft_consensus.cc:3037] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:07.097465 10801 raft_consensus.cc:2454] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200ba047bfaf481f8953be562ff2c096 in term 1.
I20260620 08:18:07.097473 10932 raft_consensus.cc:2454] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 200ba047bfaf481f8953be562ff2c096 in term 1.
I20260620 08:18:07.097664 10997 leader_election.cc:302] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200ba047bfaf481f8953be562ff2c096, 67bbfd2cc53d431a922a49f6e5ed4d58; no voters:
I20260620 08:18:07.097815 11130 raft_consensus.cc:2781] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:07.098001 11130 raft_consensus.cc:686] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 1 LEADER]: Becoming Leader. State: Replica: 200ba047bfaf481f8953be562ff2c096, State: Running, Role: LEADER
I20260620 08:18:07.098112 11130 consensus_queue.cc:237] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } }
I20260620 08:18:07.099069 10659 catalog_manager.cc:5495] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 200ba047bfaf481f8953be562ff2c096 (127.24.249.131). New cstate: current_term: 1 leader_uuid: "200ba047bfaf481f8953be562ff2c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "200ba047bfaf481f8953be562ff2c096" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 } health_report { overall_health: UNKNOWN } } }
W20260620 08:18:07.100648 10979 tablet.cc:2369] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:18:07.126863 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.124311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.24.249.131:45747)
0620 08:18:07.124380 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:07.124384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:07.124398 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:18:07.124439 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:07.124440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:07.124466 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:07.124551 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:07.124692 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.125174 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.126036 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:07.126230 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:07.126340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:07.126503 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:07.126611 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:07.126708 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:07.126728 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 08:18:07.126747 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:18:07.128926 10932 raft_consensus.cc:1261] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 1 FOLLOWER]: Refusing update from remote peer 200ba047bfaf481f8953be562ff2c096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:07.128911 10801 raft_consensus.cc:1261] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 1 FOLLOWER]: Refusing update from remote peer 200ba047bfaf481f8953be562ff2c096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:07.129271 11130 consensus_queue.cc:1035] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67bbfd2cc53d431a922a49f6e5ed4d58" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:07.129400 11133 consensus_queue.cc:1035] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b12f0d344214fe9acdac6fd88f6c2ee" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:07.132079 11136 mvcc.cc:205] Tried to move back new op lower bound from 7298840523277221888 to 7298840523154956288. Current Snapshot: MvccSnapshot[applied={T|T < 7298840523277221888}]
I20260620 08:18:07.132280 11137 mvcc.cc:205] Tried to move back new op lower bound from 7298840523277221888 to 7298840523154956288. Current Snapshot: MvccSnapshot[applied={T|T < 7298840523277221888}]
I20260620 08:18:07.134999 10658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:07.135094 10658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:07.135963 10658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9698da555d44a9cad33f839d3b2995d: Sending DeleteTablet for 3 replicas of tablet 13811a326640478abed9603e32db378b
I20260620 08:18:07.136355 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet 13811a326640478abed9603e32db378b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:07 UTC) from {username='slave'} at 127.0.0.1:43796
I20260620 08:18:07.136353 10912 tablet_service.cc:1514] Processing DeleteTablet for tablet 13811a326640478abed9603e32db378b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:07 UTC) from {username='slave'} at 127.0.0.1:36154
I20260620 08:18:07.136610 11147 tablet_replica.cc:330] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: stopping tablet replica
I20260620 08:18:07.136595 11043 tablet_service.cc:1514] Processing DeleteTablet for tablet 13811a326640478abed9603e32db378b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:07 UTC) from {username='slave'} at 127.0.0.1:35936
I20260620 08:18:07.136734 11147 raft_consensus.cc:2229] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:07.136852 11147 raft_consensus.cc:2258] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:07.136847 11148 tablet_replica.cc:330] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: stopping tablet replica
I20260620 08:18:07.137001 11148 raft_consensus.cc:2229] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:18:07.137014 11146 tablet_replica.cc:330] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee: stopping tablet replica
I20260620 08:18:07.137105 11146 raft_consensus.cc:2229] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:07.137154 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10719
I20260620 08:18:07.137207 11146 raft_consensus.cc:2258] T 13811a326640478abed9603e32db378b P 4b12f0d344214fe9acdac6fd88f6c2ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:07.139561 11147 ts_tablet_manager.cc:1905] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:07.141593 11147 ts_tablet_manager.cc:1918] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:07.141665 11147 log.cc:1196] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/13811a326640478abed9603e32db378b
I20260620 08:18:07.141963 11147 ts_tablet_manager.cc:1939] T 13811a326640478abed9603e32db378b P 67bbfd2cc53d431a922a49f6e5ed4d58: Deleting consensus metadata
I20260620 08:18:07.142063 11148 raft_consensus.cc:2258] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:07.142391 10644 catalog_manager.cc:4841] TS 67bbfd2cc53d431a922a49f6e5ed4d58 (127.24.249.130:46353): tablet 13811a326640478abed9603e32db378b (table test-table [id=b6c5114f52bd457b9ccd51516df1a858]) successfully deleted
I20260620 08:18:07.142629 11148 ts_tablet_manager.cc:1905] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:18:07.143968 10645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:43537 (error 108)
I20260620 08:18:07.144294 11148 ts_tablet_manager.cc:1918] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:07.144387 11148 log.cc:1196] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/13811a326640478abed9603e32db378b
I20260620 08:18:07.144465 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:07.144321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:43537 (local address 127.0.0.1:43812)
0620 08:18:07.144385 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:07.144424 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:43537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 08:18:07.144582 10645 catalog_manager.cc:4584] TS 4b12f0d344214fe9acdac6fd88f6c2ee (127.24.249.129:43537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13811a326640478abed9603e32db378b: Network error: Client connection negotiation failed: client connection to 127.24.249.129:43537: connect: Connection refused (error 111)
I20260620 08:18:07.144625 11148 ts_tablet_manager.cc:1939] T 13811a326640478abed9603e32db378b P 200ba047bfaf481f8953be562ff2c096: Deleting consensus metadata
I20260620 08:18:07.144898 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10850
I20260620 08:18:07.145016 10643 catalog_manager.cc:4841] TS 200ba047bfaf481f8953be562ff2c096 (127.24.249.131:45747): tablet 13811a326640478abed9603e32db378b (table test-table [id=b6c5114f52bd457b9ccd51516df1a858]) successfully deleted
I20260620 08:18:07.150950 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10981
I20260620 08:18:07.155797 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 10628
2026-06-20T08:18:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (736 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:07 dist-test-slave-65w8 krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:07 dist-test-slave-65w8 krb5kdc[11152](info): set up 2 sockets
Jun 20 08:18:07 dist-test-slave-65w8 krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260620 08:18:08.217813 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:08.345299 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41609
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:08.429248 11165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:08.429512 11165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:08.429553 11165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:08.430819 11165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:08.430887 11165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:08.430912 11165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:08.430934 11165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:08.430962 11165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:08.432437 11165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:41609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:08.432721 11165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:08.432957 11165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:08.438055 11173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:08.438040 11170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:08.438087 11165 server_base.cc:1034] running on GCE node
W20260620 08:18:08.438035 11171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:08.438519 11165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:08.438891 11165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:08.440088 11165 hybrid_clock.cc:648] HybridClock initialized: now 1781943488440067 us; error 48 us; skew 500 ppm
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:08.443042 11165 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:08.443609 11165 webserver.cc:460] Webserver started at http://127.24.249.190:43243/ using document root <none> and password file <none>
I20260620 08:18:08.443854 11165 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:08.443934 11165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:08.444093 11165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:08.445199 11165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "028413857c0b4fc5966a818d70bd7a6c"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "cc753c86451455fcc07680112eabd8d6"
server_key_iv: "cf84ec774ecb32ec555cc0d9ae5ab4dd"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.445578 11165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "028413857c0b4fc5966a818d70bd7a6c"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "cc753c86451455fcc07680112eabd8d6"
server_key_iv: "cf84ec774ecb32ec555cc0d9ae5ab4dd"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.447114 11165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:18:08.448055 11180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.448333 11165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:08.448478 11165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "028413857c0b4fc5966a818d70bd7a6c"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "cc753c86451455fcc07680112eabd8d6"
server_key_iv: "cf84ec774ecb32ec555cc0d9ae5ab4dd"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.448571 11165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:08.456213 11165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:08.458406 11165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:08.458570 11165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:08.463254 11165 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:41609
I20260620 08:18:08.463282 11232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:41609 every 8 connection(s)
I20260620 08:18:08.463682 11165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:08.464568 11233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:08.466985 11233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c: Bootstrap starting.
I20260620 08:18:08.467530 11233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:08.467765 11233 log.cc:824] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:08.468449 11233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c: No bootstrap required, opened a new log
I20260620 08:18:08.470094 11233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028413857c0b4fc5966a818d70bd7a6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41609 } }
I20260620 08:18:08.470232 11233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:08.470254 11233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028413857c0b4fc5966a818d70bd7a6c, State: Initialized, Role: FOLLOWER
I20260620 08:18:08.470400 11233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028413857c0b4fc5966a818d70bd7a6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41609 } }
I20260620 08:18:08.470484 11233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:08.470507 11233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:08.470577 11233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:08.470737 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 11165
I20260620 08:18:08.470863 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:08.471065 25574 external_mini_cluster.cc:1428] Setting key e65f16ac6f3e7fd6ea5caa3b0481f2fc
I20260620 08:18:08.471333 11233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028413857c0b4fc5966a818d70bd7a6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41609 } }
I20260620 08:18:08.471457 11233 leader_election.cc:302] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 028413857c0b4fc5966a818d70bd7a6c; no voters:
I20260620 08:18:08.471622 11233 leader_election.cc:288] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:08.471704 11237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:08.471849 11237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [term 1 LEADER]: Becoming Leader. State: Replica: 028413857c0b4fc5966a818d70bd7a6c, State: Running, Role: LEADER
I20260620 08:18:08.471871 11233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:08.471951 11237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028413857c0b4fc5966a818d70bd7a6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41609 } }
I20260620 08:18:08.472263 11237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 028413857c0b4fc5966a818d70bd7a6c. Latest consensus state: current_term: 1 leader_uuid: "028413857c0b4fc5966a818d70bd7a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028413857c0b4fc5966a818d70bd7a6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41609 } } }
I20260620 08:18:08.472322 11237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:08.473261 11245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:08.473933 11245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:08.473969 11240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "028413857c0b4fc5966a818d70bd7a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028413857c0b4fc5966a818d70bd7a6c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41609 } } }
I20260620 08:18:08.474038 11240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:08.476037 11245 catalog_manager.cc:1347] Generated new cluster ID: a05ac358a9164988b93b5b418dbf9585
I20260620 08:18:08.476289 11245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:08.483970 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:08.471576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.24.249.190:41609)
0620 08:18:08.471883 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:08.471958 (+ 75us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:08.471994 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:18:08.472126 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:08.472133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:08.472189 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:08.472344 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:08.473066 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.474958 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.476100 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.476753 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.476857 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:08.479723 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:08.479728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:08.479728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:08.479740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:08.481836 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:08.482285 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:08.482289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:08.482289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:08.482423 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:08.482636 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:08.482636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:08.482637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:08.482955 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:08.483112 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:08.483596 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 08:18:08.483740 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":65,"threads_started":1}
I20260620 08:18:08.487920 11245 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:08.488586 11245 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:08.501073 11245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c: Generated new TSK 0
I20260620 08:18:08.501295 11245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:18:08.539247 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41609
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:08.630033 11259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:08.630249 11259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:08.630273 11259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:08.631466 11259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:08.631510 11259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:08.631537 11259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:18:08.633004 11259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:08.633340 11259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:08.633617 11259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:08.639219 11264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:08.639213 11267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:08.639219 11265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:08.639297 11259 server_base.cc:1034] running on GCE node
I20260620 08:18:08.639679 11259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:08.639988 11259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:08.641172 11259 hybrid_clock.cc:648] HybridClock initialized: now 1781943488641143 us; error 62 us; skew 500 ppm
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:08.644296 11259 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:18:08.644786 11259 webserver.cc:460] Webserver started at http://127.24.249.129:38851/ using document root <none> and password file <none>
I20260620 08:18:08.645063 11259 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:08.645138 11259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:08.645294 11259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:08.646458 11259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "0b16645713124b5f8fed1fb2a5347a8f"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "d402682410d25c55e34c46ea3c9e1314"
server_key_iv: "f24aca81dc0dfba761a265c2254428f0"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.646831 11259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "0b16645713124b5f8fed1fb2a5347a8f"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "d402682410d25c55e34c46ea3c9e1314"
server_key_iv: "f24aca81dc0dfba761a265c2254428f0"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.648299 11259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:08.649207 11274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.649554 11259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:08.649701 11259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "0b16645713124b5f8fed1fb2a5347a8f"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "d402682410d25c55e34c46ea3c9e1314"
server_key_iv: "f24aca81dc0dfba761a265c2254428f0"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.649806 11259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:08.660447 11259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:08.662971 11259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:08.663158 11259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:08.663456 11259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:08.663801 11259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:08.663859 11259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.663893 11259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:08.663914 11259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.670487 11259 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:40507
I20260620 08:18:08.670569 11387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:40507 every 8 connection(s)
I20260620 08:18:08.670924 11259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:18:08.675498 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 11259
I20260620 08:18:08.675650 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:18:08.675932 25574 external_mini_cluster.cc:1428] Setting key fe28420e3af8767fc9666cc016b4393e
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:08.681974 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:08.671795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:40711 (local address 127.24.249.190:41609)
0620 08:18:08.671884 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:08.671887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:08.672205 (+ 318us) server_negotiation.cc:407] Connection header received
0620 08:18:08.673373 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:08.673375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:08.673394 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:08.673463 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:08.674197 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.674718 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.675834 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.676208 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.676291 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:08.678725 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:08.678726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:08.678727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:08.678734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:08.680357 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:08.680853 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:08.680854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:08.680855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:08.680928 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:08.681214 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:08.681215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:08.681216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:08.681357 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:08.681442 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:08.681747 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 08:18:08.681844 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 08:18:08.682238 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:08.671774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41609 (local address 127.24.249.129:40711)
0620 08:18:08.672051 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:08.672090 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:08.673248 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:08.673559 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:08.673563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:08.673837 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:08.674102 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:08.674107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.674826 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.674827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:08.675719 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:08.675721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.676325 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.676326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:08.676473 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:08.677061 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:08.677070 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:08.678603 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:08.680490 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:08.680498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:08.680501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:08.680764 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:08.681008 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:08.681010 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:08.681010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:08.681137 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:08.681432 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:08.681440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:08.681600 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 08:18:08.681723 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 08:18:08.681947 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":149,"threads_started":1}
I20260620 08:18:08.682763 11388 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41609
I20260620 08:18:08.682865 11388 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:08.683084 11388 heartbeater.cc:511] Master 127.24.249.190:41609 requested a full tablet report, sending...
I20260620 08:18:08.683676 11197 ts_manager.cc:194] Registered new tserver with Master: 0b16645713124b5f8fed1fb2a5347a8f (127.24.249.129:40507)
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:18:08.684365 11197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:40711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:18:08.704917 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41609
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:08.787706 11393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:08.787914 11393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:08.787937 11393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:08.789093 11393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:08.789136 11393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:08.789163 11393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:18:08.790654 11393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:08.790887 11393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:08.791145 11393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:18:08.796365 11393 server_base.cc:1034] running on GCE node
W20260620 08:18:08.796348 11399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:08.796347 11398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:08.796360 11401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:08.796881 11393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:08.797140 11393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:08.798326 11393 hybrid_clock.cc:648] HybridClock initialized: now 1781943488798303 us; error 37 us; skew 500 ppm
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:08.800951 11393 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:18:08.801379 11393 webserver.cc:460] Webserver started at http://127.24.249.130:38037/ using document root <none> and password file <none>
I20260620 08:18:08.801569 11393 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:08.801610 11393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:08.801700 11393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:08.802775 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "26c07143c74a41fa82d14223a7267d67"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "0529f4c68400b6e9fe2317c11f9e4d03"
server_key_iv: "25f86fde23c0e66b8c981e9e77300e5b"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.803086 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "26c07143c74a41fa82d14223a7267d67"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "0529f4c68400b6e9fe2317c11f9e4d03"
server_key_iv: "25f86fde23c0e66b8c981e9e77300e5b"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.804448 11393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:08.805250 11408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.805444 11393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:08.805553 11393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "26c07143c74a41fa82d14223a7267d67"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "0529f4c68400b6e9fe2317c11f9e4d03"
server_key_iv: "25f86fde23c0e66b8c981e9e77300e5b"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.805629 11393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:08.817268 11393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:08.819438 11393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:08.819613 11393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:08.819854 11393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:08.820237 11393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:08.820295 11393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.820338 11393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:08.820375 11393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.826278 11393 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:39441
I20260620 08:18:08.826361 11521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:39441 every 8 connection(s)
I20260620 08:18:08.826696 11393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:18:08.830313 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 11393
I20260620 08:18:08.830437 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:18:08.830655 25574 external_mini_cluster.cc:1428] Setting key 2f03deecae2a9cc3d4093deb35b46729
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:08.838438 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:08.827479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:50547 (local address 127.24.249.190:41609)
0620 08:18:08.827550 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:08.827553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:08.827766 (+ 213us) server_negotiation.cc:407] Connection header received
0620 08:18:08.828410 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:08.828411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:08.828422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:08.828520 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:08.829529 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.830213 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.831691 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.831992 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.832071 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:08.835095 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:08.835096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:08.835096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:08.835104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:08.836755 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:08.837339 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:08.837340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:08.837340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:08.837393 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:08.837728 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:08.837729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:08.837729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:08.837836 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:08.837948 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:08.838300 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 08:18:08.838368 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:18:08.838766 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:08.827453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41609 (local address 127.24.249.130:50547)
0620 08:18:08.827644 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:08.827682 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:08.828162 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:08.828606 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:08.828610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:08.829188 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:08.829386 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:08.829391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.830339 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.830340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:08.831607 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:08.831611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:08.832090 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:08.832091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:08.832209 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:08.832891 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:08.832904 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:08.834981 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:08.836858 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:08.836868 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:08.836871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:08.837251 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:08.837470 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:08.837471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:08.837471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:08.837646 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:08.837939 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:08.837946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:08.838137 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 08:18:08.838267 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 08:18:08.838483 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260620 08:18:08.839380 11522 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41609
I20260620 08:18:08.839493 11522 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:08.839713 11522 heartbeater.cc:511] Master 127.24.249.190:41609 requested a full tablet report, sending...
I20260620 08:18:08.840205 11197 ts_manager.cc:194] Registered new tserver with Master: 26c07143c74a41fa82d14223a7267d67 (127.24.249.130:39441)
I20260620 08:18:08.840581 11197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:50547
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:18:08.861095 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41609
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:08.943459 11527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:08.943724 11527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:08.943764 11527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:08.944945 11527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:08.945021 11527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:08.945068 11527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:18:08.946624 11527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:41609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:08.946909 11527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:08.947146 11527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:08.952517 11532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:08.952512 11535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:08.952600 11527 server_base.cc:1034] running on GCE node
W20260620 08:18:08.952549 11533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:08.953140 11527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:08.953368 11527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:08.954595 11527 hybrid_clock.cc:648] HybridClock initialized: now 1781943488954574 us; error 39 us; skew 500 ppm
Jun 20 08:18:08 dist-test-slave-65w8 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:08.957551 11527 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:18:08.958212 11527 webserver.cc:460] Webserver started at http://127.24.249.131:33391/ using document root <none> and password file <none>
I20260620 08:18:08.958470 11527 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:08.958539 11527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:08.958678 11527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:08.960112 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "a0aecea670c84cf0b0bdf11cf22188f8"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "4de6d1b25f4a851cc74952c6c107b813"
server_key_iv: "73ae57301dbccd525d726d892ba5ab89"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.960601 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "a0aecea670c84cf0b0bdf11cf22188f8"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "4de6d1b25f4a851cc74952c6c107b813"
server_key_iv: "73ae57301dbccd525d726d892ba5ab89"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.962141 11527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 08:18:08.963037 11542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:08.963253 11527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:08.963361 11527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "a0aecea670c84cf0b0bdf11cf22188f8"
format_stamp: "Formatted at 2026-06-20 08:18:08 on dist-test-slave-65w8"
server_key: "4de6d1b25f4a851cc74952c6c107b813"
server_key_iv: "73ae57301dbccd525d726d892ba5ab89"
server_key_version: "encryptionkey@0"
I20260620 08:18:08.963471 11527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:08.998029 11527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:09.000849 11527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:09.001052 11527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:09.001353 11527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:09.001704 11527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:09.001765 11527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.001799 11527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:09.001830 11527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.008150 11527 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:42633
I20260620 08:18:09.008217 11655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:42633 every 8 connection(s)
I20260620 08:18:09.008565 11527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
Jun 20 08:18:09 dist-test-slave-65w8 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943488, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:09.017179 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 11527
I20260620 08:18:09.017287 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:18:09.017511 25574 external_mini_cluster.cc:1428] Setting key 67ccfb987560af36ed6378eceb2d9239
I20260620 08:18:09.019965 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.009328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:54813 (local address 127.24.249.190:41609)
0620 08:18:09.009408 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.009411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.009650 (+ 239us) server_negotiation.cc:407] Connection header received
0620 08:18:09.010297 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.010299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.010310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.010390 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:09.011537 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.012056 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.013306 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.013570 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.013636 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.016321 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:09.016323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:09.016323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:09.016331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:09.018216 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:09.018743 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:09.018750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:09.018751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:09.018816 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:09.019173 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:09.019174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:09.019174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:09.019328 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:09.019416 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.019799 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.019896 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 08:18:09.020102 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.009279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:41609 (local address 127.24.249.131:54813)
0620 08:18:09.009476 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.009523 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.010142 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.010501 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.010505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.011156 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:09.011414 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.011419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.012162 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.012163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.013171 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.013174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.013680 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.013681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.013808 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.014526 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:09.014535 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:09.016201 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:09.018328 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:09.018336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:09.018338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:09.018664 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:09.018945 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:09.018946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:09.018946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:09.019075 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:09.019430 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:09.019437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:09.019617 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.019723 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.019868 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":108,"threads_started":1}
I20260620 08:18:09.020766 11656 heartbeater.cc:347] Connected to a master server at 127.24.249.190:41609
I20260620 08:18:09.020862 11656 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:09.021015 11656 heartbeater.cc:511] Master 127.24.249.190:41609 requested a full tablet report, sending...
I20260620 08:18:09.021379 11197 ts_manager.cc:194] Registered new tserver with Master: a0aecea670c84cf0b0bdf11cf22188f8 (127.24.249.131:42633)
I20260620 08:18:09.021783 11197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:54813
I20260620 08:18:09.029644 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:18:09.037528 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.031036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.24.249.190:41609)
0620 08:18:09.031106 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.031108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.031227 (+ 119us) server_negotiation.cc:407] Connection header received
0620 08:18:09.031269 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.031269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.031278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.031333 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:09.031873 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.032358 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.033113 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.033355 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.033420 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.034861 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:09.034862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:09.034863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:09.034870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:09.036331 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:09.036795 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:09.036796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:09.036796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:09.036852 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:09.037053 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:09.037054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:09.037054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:09.037188 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:09.037267 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.037411 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.037463 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:18:09.039184 11197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:18:09.040694 11197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:18:09.047601 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.043116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:42633 (local address 127.0.0.1:41846)
0620 08:18:09.043565 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.043582 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.043653 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.044166 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.044167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.044169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.044242 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.044244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.045168 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.045169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.046544 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.046546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.047266 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.047267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.047350 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.047364 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.047432 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.047503 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":8,"thread_start_us":101,"threads_started":1}
I20260620 08:18:09.047684 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.043199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.24.249.130:39441)
0620 08:18:09.043410 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.043416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.043541 (+ 125us) server_negotiation.cc:407] Connection header received
0620 08:18:09.043728 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.043731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.043760 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.043861 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.044067 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.045273 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.046608 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.047292 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.047361 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.047412 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.047548 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.047596 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260620 08:18:09.047704 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.043220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.24.249.131:42633)
0620 08:18:09.043590 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.043594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.043638 (+ 44us) server_negotiation.cc:407] Connection header received
0620 08:18:09.043983 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.043986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.044002 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.044082 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.044327 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.045060 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.046652 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.047172 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.047270 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.047331 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.047575 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.047619 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260620 08:18:09.047933 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.043628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.24.249.129:40507)
0620 08:18:09.043894 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.043898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.044065 (+ 167us) server_negotiation.cc:407] Connection header received
0620 08:18:09.044109 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.044111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.044127 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.044249 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.044359 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.045422 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.046548 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.047286 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.047361 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.047412 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.047814 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.047859 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":80,"threads_started":1}
I20260620 08:18:09.047734 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.043373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:40507 (local address 127.0.0.1:35346)
0620 08:18:09.043942 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.043953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.044006 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.044260 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.044261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.044262 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.044298 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.044300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.045525 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.045526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.046459 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.046462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.047448 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.047449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.047562 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.047567 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.047614 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.047653 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260620 08:18:09.047909 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.043084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:39441 (local address 127.0.0.1:58198)
0620 08:18:09.043428 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.043472 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.043588 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.043888 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.043891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.043894 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.043978 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.043980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.045405 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.045406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.046530 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.046533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.047380 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.047380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.047455 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.047457 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.047521 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.047854 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":275,"thread_start_us":66,"threads_started":1}
I20260620 08:18:09.048934 11322 tablet_service.cc:1467] Processing CreateTablet for tablet d388b43b423a4a5682924d7e32fcba92 (DEFAULT_TABLE table=test-table [id=4fa87d7529dc41ef93cdb8c5925f05d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:09.048949 11456 tablet_service.cc:1467] Processing CreateTablet for tablet d388b43b423a4a5682924d7e32fcba92 (DEFAULT_TABLE table=test-table [id=4fa87d7529dc41ef93cdb8c5925f05d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:09.048949 11590 tablet_service.cc:1467] Processing CreateTablet for tablet d388b43b423a4a5682924d7e32fcba92 (DEFAULT_TABLE table=test-table [id=4fa87d7529dc41ef93cdb8c5925f05d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:09.049232 11322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d388b43b423a4a5682924d7e32fcba92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:09.049271 11590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d388b43b423a4a5682924d7e32fcba92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:09.049278 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d388b43b423a4a5682924d7e32fcba92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:09.052280 11673 tablet_bootstrap.cc:492] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f: Bootstrap starting.
I20260620 08:18:09.052500 11674 tablet_bootstrap.cc:492] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Bootstrap starting.
I20260620 08:18:09.053179 11673 tablet_bootstrap.cc:654] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:09.053179 11674 tablet_bootstrap.cc:654] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:09.053876 11673 log.cc:824] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:09.053882 11674 log.cc:824] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:09.054039 11675 tablet_bootstrap.cc:492] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Bootstrap starting.
I20260620 08:18:09.054708 11675 tablet_bootstrap.cc:654] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:09.054852 11674 tablet_bootstrap.cc:492] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: No bootstrap required, opened a new log
I20260620 08:18:09.054924 11674 ts_tablet_manager.cc:1397] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:18:09.055248 11673 tablet_bootstrap.cc:492] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f: No bootstrap required, opened a new log
I20260620 08:18:09.055318 11673 ts_tablet_manager.cc:1397] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:18:09.055371 11675 log.cc:824] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:09.056078 11675 tablet_bootstrap.cc:492] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: No bootstrap required, opened a new log
I20260620 08:18:09.056164 11675 ts_tablet_manager.cc:1397] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:18:09.056990 11674 raft_consensus.cc:348] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.057196 11674 raft_consensus.cc:374] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:09.057235 11674 raft_consensus.cc:729] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c07143c74a41fa82d14223a7267d67, State: Initialized, Role: FOLLOWER
I20260620 08:18:09.057366 11674 consensus_queue.cc:260] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.057394 11673 raft_consensus.cc:348] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.057581 11673 raft_consensus.cc:374] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:09.057626 11673 raft_consensus.cc:729] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b16645713124b5f8fed1fb2a5347a8f, State: Initialized, Role: FOLLOWER
I20260620 08:18:09.057691 11674 ts_tablet_manager.cc:1428] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:09.057816 11522 heartbeater.cc:503] Master 127.24.249.190:41609 was elected leader, sending a full tablet report...
I20260620 08:18:09.057756 11675 raft_consensus.cc:348] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.057965 11675 raft_consensus.cc:374] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:09.058012 11675 raft_consensus.cc:729] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0aecea670c84cf0b0bdf11cf22188f8, State: Initialized, Role: FOLLOWER
I20260620 08:18:09.058121 11675 consensus_queue.cc:260] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.057768 11673 consensus_queue.cc:260] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.058393 11675 ts_tablet_manager.cc:1428] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:18:09.058420 11656 heartbeater.cc:503] Master 127.24.249.190:41609 was elected leader, sending a full tablet report...
I20260620 08:18:09.058959 11388 heartbeater.cc:503] Master 127.24.249.190:41609 was elected leader, sending a full tablet report...
I20260620 08:18:09.058815 11673 ts_tablet_manager.cc:1428] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 08:18:09.077344 11523 tablet.cc:2369] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:18:09.171896 11389 tablet.cc:2369] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:18:09.188385 11681 raft_consensus.cc:482] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:18:09.188529 11681 raft_consensus.cc:504] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.189044 11681 leader_election.cc:288] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c07143c74a41fa82d14223a7267d67 (127.24.249.130:39441), a0aecea670c84cf0b0bdf11cf22188f8 (127.24.249.131:42633)
I20260620 08:18:09.192008 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.189304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:42003 (local address 127.24.249.130:39441)
0620 08:18:09.189352 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.189354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.189363 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:09.189424 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.189424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.189434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.189487 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.189652 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.190222 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.191236 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.191630 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.191723 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.191733 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.191895 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.191942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:18:09.192021 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.189169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:39441 (local address 127.24.249.129:42003)
0620 08:18:09.189284 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.189297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.189356 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.189490 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.189491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.189493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.189568 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.189571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.190313 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.190314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.191133 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.191135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.191723 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.191724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.191799 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.191803 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.191876 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.191925 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260620 08:18:09.192373 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d388b43b423a4a5682924d7e32fcba92" candidate_uuid: "0b16645713124b5f8fed1fb2a5347a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c07143c74a41fa82d14223a7267d67" is_pre_election: true
I20260620 08:18:09.192507 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.189328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:35715 (local address 127.24.249.131:42633)
0620 08:18:09.189399 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.189402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.189506 (+ 104us) server_negotiation.cc:407] Connection header received
0620 08:18:09.189599 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.189600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.189617 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.189694 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.189904 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.190428 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.191536 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.192036 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.192144 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.192155 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.192394 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.192435 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 08:18:09.192531 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.189261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:42633 (local address 127.24.249.129:35715)
0620 08:18:09.189417 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.189427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.189520 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.189697 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.189698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.189700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.189793 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.189795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.190537 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.190539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.191425 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.191427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.192220 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.192222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.192307 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.192311 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.192389 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.192434 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":6,"thread_start_us":75,"threads_started":1}
I20260620 08:18:09.192571 11476 raft_consensus.cc:2454] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b16645713124b5f8fed1fb2a5347a8f in term 0.
I20260620 08:18:09.192911 11610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d388b43b423a4a5682924d7e32fcba92" candidate_uuid: "0b16645713124b5f8fed1fb2a5347a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" is_pre_election: true
I20260620 08:18:09.193001 11277 leader_election.cc:302] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b16645713124b5f8fed1fb2a5347a8f, 26c07143c74a41fa82d14223a7267d67; no voters:
I20260620 08:18:09.193077 11610 raft_consensus.cc:2454] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b16645713124b5f8fed1fb2a5347a8f in term 0.
I20260620 08:18:09.193173 11681 raft_consensus.cc:2781] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:18:09.193252 11681 raft_consensus.cc:482] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:18:09.193316 11681 raft_consensus.cc:3037] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:09.194196 11681 raft_consensus.cc:504] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.194351 11681 leader_election.cc:288] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [CANDIDATE]: Term 1 election: Requested vote from peers 26c07143c74a41fa82d14223a7267d67 (127.24.249.130:39441), a0aecea670c84cf0b0bdf11cf22188f8 (127.24.249.131:42633)
I20260620 08:18:09.194525 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d388b43b423a4a5682924d7e32fcba92" candidate_uuid: "0b16645713124b5f8fed1fb2a5347a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c07143c74a41fa82d14223a7267d67"
I20260620 08:18:09.194581 11610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d388b43b423a4a5682924d7e32fcba92" candidate_uuid: "0b16645713124b5f8fed1fb2a5347a8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0aecea670c84cf0b0bdf11cf22188f8"
I20260620 08:18:09.194648 11610 raft_consensus.cc:3037] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:09.194650 11476 raft_consensus.cc:3037] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:09.195545 11476 raft_consensus.cc:2454] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b16645713124b5f8fed1fb2a5347a8f in term 1.
I20260620 08:18:09.195549 11610 raft_consensus.cc:2454] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b16645713124b5f8fed1fb2a5347a8f in term 1.
I20260620 08:18:09.195765 11277 leader_election.cc:302] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b16645713124b5f8fed1fb2a5347a8f, 26c07143c74a41fa82d14223a7267d67; no voters:
I20260620 08:18:09.195868 11681 raft_consensus.cc:2781] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:09.196065 11681 raft_consensus.cc:686] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [term 1 LEADER]: Becoming Leader. State: Replica: 0b16645713124b5f8fed1fb2a5347a8f, State: Running, Role: LEADER
I20260620 08:18:09.196148 11681 consensus_queue.cc:237] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } }
I20260620 08:18:09.197211 11197 catalog_manager.cc:5495] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b16645713124b5f8fed1fb2a5347a8f (127.24.249.129). New cstate: current_term: 1 leader_uuid: "0b16645713124b5f8fed1fb2a5347a8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b16645713124b5f8fed1fb2a5347a8f" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 40507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 } health_report { overall_health: UNKNOWN } } }
I20260620 08:18:09.230059 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.227558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.24.249.129:40507)
0620 08:18:09.227628 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.227629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.227636 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:18:09.227672 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.227673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.227691 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.227772 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:09.227916 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.228394 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.229180 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.229381 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.229472 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.229672 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:09.229763 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:09.229865 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.229968 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.229983 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 08:18:09.232156 11610 raft_consensus.cc:1261] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 1 FOLLOWER]: Refusing update from remote peer 0b16645713124b5f8fed1fb2a5347a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:09.232156 11476 raft_consensus.cc:1261] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 1 FOLLOWER]: Refusing update from remote peer 0b16645713124b5f8fed1fb2a5347a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:09.232545 11684 consensus_queue.cc:1035] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c07143c74a41fa82d14223a7267d67" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:09.232671 11681 consensus_queue.cc:1035] T d388b43b423a4a5682924d7e32fcba92 P 0b16645713124b5f8fed1fb2a5347a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0aecea670c84cf0b0bdf11cf22188f8" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:09.235550 11689 mvcc.cc:205] Tried to move back new op lower bound from 7298840531892146176 to 7298840531748245504. Current Snapshot: MvccSnapshot[applied={T|T < 7298840531892146176}]
I20260620 08:18:09.237805 11196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:09.237947 11196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:09.238867 11196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 028413857c0b4fc5966a818d70bd7a6c: Sending DeleteTablet for 3 replicas of tablet d388b43b423a4a5682924d7e32fcba92
I20260620 08:18:09.239248 11456 tablet_service.cc:1514] Processing DeleteTablet for tablet d388b43b423a4a5682924d7e32fcba92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:09 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:58198
I20260620 08:18:09.239503 11322 tablet_service.cc:1514] Processing DeleteTablet for tablet d388b43b423a4a5682924d7e32fcba92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:09 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:35346
I20260620 08:18:09.239557 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 11259
I20260620 08:18:09.240231 11590 tablet_service.cc:1514] Processing DeleteTablet for tablet d388b43b423a4a5682924d7e32fcba92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:09 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:41846
I20260620 08:18:09.240278 11697 tablet_replica.cc:330] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: stopping tablet replica
I20260620 08:18:09.240381 11697 raft_consensus.cc:2229] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:09.240484 11697 raft_consensus.cc:2258] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:09.240885 11697 ts_tablet_manager.cc:1905] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:09.242146 11698 tablet_replica.cc:330] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: stopping tablet replica
I20260620 08:18:09.242234 11697 ts_tablet_manager.cc:1918] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:09.242304 11697 log.cc:1196] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/d388b43b423a4a5682924d7e32fcba92
I20260620 08:18:09.242326 11698 raft_consensus.cc:2229] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:09.242472 11698 raft_consensus.cc:2258] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:09.242604 11697 ts_tablet_manager.cc:1939] T d388b43b423a4a5682924d7e32fcba92 P 26c07143c74a41fa82d14223a7267d67: Deleting consensus metadata
I20260620 08:18:09.242908 11698 ts_tablet_manager.cc:1905] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:09.243050 11183 catalog_manager.cc:4841] TS 26c07143c74a41fa82d14223a7267d67 (127.24.249.130:39441): tablet d388b43b423a4a5682924d7e32fcba92 (table test-table [id=4fa87d7529dc41ef93cdb8c5925f05d6]) successfully deleted
I20260620 08:18:09.244221 11698 ts_tablet_manager.cc:1918] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:09.244311 11698 log.cc:1196] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/d388b43b423a4a5682924d7e32fcba92
I20260620 08:18:09.244545 11698 ts_tablet_manager.cc:1939] T d388b43b423a4a5682924d7e32fcba92 P a0aecea670c84cf0b0bdf11cf22188f8: Deleting consensus metadata
I20260620 08:18:09.245002 11181 catalog_manager.cc:4841] TS a0aecea670c84cf0b0bdf11cf22188f8 (127.24.249.131:42633): tablet d388b43b423a4a5682924d7e32fcba92 (table test-table [id=4fa87d7529dc41ef93cdb8c5925f05d6]) successfully deleted
W20260620 08:18:09.246788 11183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:40507 (error 108)
I20260620 08:18:09.246987 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 11393
I20260620 08:18:09.247903 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.247764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:40507 (local address 127.0.0.1:35370)
0620 08:18:09.247823 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.247861 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 08:18:09.248039 11183 catalog_manager.cc:4584] TS 0b16645713124b5f8fed1fb2a5347a8f (127.24.249.129:40507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d388b43b423a4a5682924d7e32fcba92: Network error: Client connection negotiation failed: client connection to 127.24.249.129:40507: connect: Connection refused (error 111)
I20260620 08:18:09.252957 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 11527
I20260620 08:18:09.257489 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 11165
2026-06-20T08:18:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2102 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T08:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:09Z Disabled control of system clock
I20260620 08:18:09.286804 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:39175
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:39175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:09.369290 11703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:09.369560 11703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:09.369581 11703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:09.371054 11703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:09.371110 11703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:09.371124 11703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:09.371136 11703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:09.371150 11703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:09.373023 11703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:39175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:39175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:09.373337 11703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:09.373704 11703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:09.378715 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.378783 11708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.378731 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:09.378960 11703 server_base.cc:1034] running on GCE node
I20260620 08:18:09.379237 11703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:09.379524 11703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:09.380702 11703 hybrid_clock.cc:648] HybridClock initialized: now 1781943489380693 us; error 40 us; skew 500 ppm
I20260620 08:18:09.382047 11703 webserver.cc:460] Webserver started at http://127.24.249.190:45311/ using document root <none> and password file <none>
I20260620 08:18:09.382293 11703 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:09.382362 11703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:09.382525 11703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:09.383608 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "1d511fa8180140f29e5330a453812e30"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "65ae66912468604f821c9423977ae7d1"
server_key_iv: "6181c4f81a366ff2963762978314e4d3"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.383975 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "1d511fa8180140f29e5330a453812e30"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "65ae66912468604f821c9423977ae7d1"
server_key_iv: "6181c4f81a366ff2963762978314e4d3"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.385278 11703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:09.386164 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.386370 11703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:09.386479 11703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "1d511fa8180140f29e5330a453812e30"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "65ae66912468604f821c9423977ae7d1"
server_key_iv: "6181c4f81a366ff2963762978314e4d3"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.386566 11703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:09.399751 11703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:09.400146 11703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:09.400297 11703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:09.404277 11703 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:39175
I20260620 08:18:09.404310 11769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:39175 every 8 connection(s)
I20260620 08:18:09.404654 11703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:09.405649 11770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:09.407940 11770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30: Bootstrap starting.
I20260620 08:18:09.408545 11770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:09.408851 11770 log.cc:824] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:09.409559 11770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30: No bootstrap required, opened a new log
I20260620 08:18:09.411197 11770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d511fa8180140f29e5330a453812e30" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39175 } }
I20260620 08:18:09.411334 11770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:09.411355 11770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d511fa8180140f29e5330a453812e30, State: Initialized, Role: FOLLOWER
I20260620 08:18:09.411506 11770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d511fa8180140f29e5330a453812e30" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39175 } }
I20260620 08:18:09.411587 11770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:09.411630 11770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:09.411684 11770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:09.411682 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 11703
I20260620 08:18:09.411785 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:09.411998 25574 external_mini_cluster.cc:1428] Setting key 4f844cbb0e424a65a836be09bd50cdfb
I20260620 08:18:09.412379 11770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d511fa8180140f29e5330a453812e30" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39175 } }
I20260620 08:18:09.412598 11770 leader_election.cc:302] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d511fa8180140f29e5330a453812e30; no voters:
I20260620 08:18:09.412829 11770 leader_election.cc:288] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:09.413005 11776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:09.413096 11770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:09.413157 11776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [term 1 LEADER]: Becoming Leader. State: Replica: 1d511fa8180140f29e5330a453812e30, State: Running, Role: LEADER
I20260620 08:18:09.413295 11776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d511fa8180140f29e5330a453812e30" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39175 } }
I20260620 08:18:09.413910 11777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d511fa8180140f29e5330a453812e30. Latest consensus state: current_term: 1 leader_uuid: "1d511fa8180140f29e5330a453812e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d511fa8180140f29e5330a453812e30" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39175 } } }
I20260620 08:18:09.413986 11777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:09.413803 11775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d511fa8180140f29e5330a453812e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d511fa8180140f29e5330a453812e30" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 39175 } } }
I20260620 08:18:09.414309 11775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30 [sys.catalog]: This master's current role is: LEADER
W20260620 08:18:09.414911 11792 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:09.414974 11792 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:09.415055 11784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:09.415294 11784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:09.416883 11784 catalog_manager.cc:1347] Generated new cluster ID: 845081a304f94fdb9335183d9213c619
I20260620 08:18:09.416957 11784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:09.423581 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.412636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.24.249.190:39175)
0620 08:18:09.412892 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.412901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.412943 (+ 42us) server_negotiation.cc:407] Connection header received
0620 08:18:09.413084 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.413092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.413165 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.413597 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:09.413709 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.415272 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.416299 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.421972 (+ 5673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.422099 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.422868 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:09.422872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:09.422873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:09.423076 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:09.423115 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:09.423128 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:09.423188 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.423298 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.423375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":132,"threads_started":1}
I20260620 08:18:09.432145 11784 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:09.432698 11784 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:09.439684 11784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30: Generated new TSK 0
I20260620 08:18:09.439962 11784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:09.448964 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39175
--builtin_ntp_servers=127.24.249.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:09.541795 11795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:09.542320 11795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:09.542384 11795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:09.543897 11795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:09.543953 11795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:09.543985 11795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:18:09.545625 11795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:09.545961 11795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:09.546332 11795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:09.551976 11801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.551985 11800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.551999 11803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:09.552371 11795 server_base.cc:1034] running on GCE node
I20260620 08:18:09.552639 11795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:09.552906 11795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:09.554075 11795 hybrid_clock.cc:648] HybridClock initialized: now 1781943489554053 us; error 38 us; skew 500 ppm
I20260620 08:18:09.560914 11795 webserver.cc:460] Webserver started at http://127.24.249.129:37845/ using document root <none> and password file <none>
I20260620 08:18:09.561239 11795 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:09.561333 11795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:09.561506 11795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:09.562781 11795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "a95fe2a777104113bb95a3fd8a4113ac"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "42cde3b416bbd464c802105c2c3df97c"
server_key_iv: "443b3d831bebfe3dc65514b09ecae398"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.563195 11795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "a95fe2a777104113bb95a3fd8a4113ac"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "42cde3b416bbd464c802105c2c3df97c"
server_key_iv: "443b3d831bebfe3dc65514b09ecae398"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.564829 11795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:09.565949 11809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.566140 11795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:09.566234 11795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "a95fe2a777104113bb95a3fd8a4113ac"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "42cde3b416bbd464c802105c2c3df97c"
server_key_iv: "443b3d831bebfe3dc65514b09ecae398"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.566330 11795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:09.580394 11795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:09.580766 11795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:09.580935 11795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:09.581187 11795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:09.581561 11795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:09.581621 11795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.581667 11795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:09.581698 11795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.588238 11795 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:36479
I20260620 08:18:09.588321 11922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:36479 every 8 connection(s)
I20260620 08:18:09.588716 11795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:18:09.594959 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.589656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:57921 (local address 127.24.249.190:39175)
0620 08:18:09.589743 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.589746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.590062 (+ 316us) server_negotiation.cc:407] Connection header received
0620 08:18:09.590756 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.590757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.590774 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.590846 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:09.591678 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.592196 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.593237 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.593481 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.593543 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.594600 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:09.594602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:09.594602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:09.594608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:09.594626 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:09.594631 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:09.594710 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.594824 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.594864 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 08:18:09.595232 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.589620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39175 (local address 127.24.249.129:57921)
0620 08:18:09.589929 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.589983 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.590625 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.590960 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.590964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.590973 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:09.591552 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.591557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.592293 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.592294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.593142 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.593144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.593590 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.593591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.593730 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.594377 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:09.594378 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:09.594442 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:09.594475 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:09.594713 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:09.594721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:09.594724 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.594828 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.594947 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260620 08:18:09.595819 11923 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39175
I20260620 08:18:09.595935 11923 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:09.596148 11923 heartbeater.cc:511] Master 127.24.249.190:39175 requested a full tablet report, sending...
I20260620 08:18:09.596135 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 11795
I20260620 08:18:09.596247 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:18:09.596556 25574 external_mini_cluster.cc:1428] Setting key 68e7c99e3c91fe4ee2283a760617d356
I20260620 08:18:09.596714 11734 ts_manager.cc:194] Registered new tserver with Master: a95fe2a777104113bb95a3fd8a4113ac (127.24.249.129:36479)
I20260620 08:18:09.597743 11734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.129:57921
I20260620 08:18:09.597764 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39175
--builtin_ntp_servers=127.24.249.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:09.682802 11926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:09.683045 11926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:09.683071 11926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:09.684357 11926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:09.684407 11926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:09.684439 11926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:18:09.685940 11926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:09.686215 11926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:09.686579 11926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:09.692138 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.692176 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.692150 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:09.692423 11926 server_base.cc:1034] running on GCE node
I20260620 08:18:09.692678 11926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:09.692955 11926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:09.694139 11926 hybrid_clock.cc:648] HybridClock initialized: now 1781943489694136 us; error 44 us; skew 500 ppm
I20260620 08:18:09.695600 11926 webserver.cc:460] Webserver started at http://127.24.249.130:37929/ using document root <none> and password file <none>
I20260620 08:18:09.695858 11926 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:09.695933 11926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:09.696096 11926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:09.697214 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "d7af81d3ea53422ab7096cb89228daee"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "edce30b5416a3224382c642884cdab44"
server_key_iv: "d8719e296429cfa71b68e6c3d516d879"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.697611 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "d7af81d3ea53422ab7096cb89228daee"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "edce30b5416a3224382c642884cdab44"
server_key_iv: "d8719e296429cfa71b68e6c3d516d879"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.699029 11926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:09.699898 11940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.700125 11926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:09.700229 11926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "d7af81d3ea53422ab7096cb89228daee"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "edce30b5416a3224382c642884cdab44"
server_key_iv: "d8719e296429cfa71b68e6c3d516d879"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.700318 11926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:09.720947 11926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:09.721313 11926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:09.721484 11926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:09.721740 11926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:09.722177 11926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:09.722235 11926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.722272 11926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:09.722318 11926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.728259 11926 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:39257
I20260620 08:18:09.728312 12053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:39257 every 8 connection(s)
I20260620 08:18:09.728655 11926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:18:09.733487 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 11926
I20260620 08:18:09.733621 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:18:09.733916 25574 external_mini_cluster.cc:1428] Setting key c7e41a9f6b40180e12064e02aee7816e
I20260620 08:18:09.734941 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.729470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:44501 (local address 127.24.249.190:39175)
0620 08:18:09.729571 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.729573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.729896 (+ 323us) server_negotiation.cc:407] Connection header received
0620 08:18:09.730424 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.730425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.730438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.730505 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:09.731181 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.731729 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.732800 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.733060 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.733280 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.734467 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:09.734468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:09.734468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:09.734474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:09.734493 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:09.734498 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:09.734572 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.734765 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.734821 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 08:18:09.735211 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.729454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39175 (local address 127.24.249.130:44501)
0620 08:18:09.729674 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.729712 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.730256 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.730565 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.730568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.730577 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:09.731089 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.731094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.731823 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.731824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.732698 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.732701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.733299 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.733300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.733420 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.734252 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:09.734253 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:09.734298 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:09.734330 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:09.734629 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:09.734636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:09.734639 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.734762 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.734937 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":129,"threads_started":1}
I20260620 08:18:09.735220 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39175
--builtin_ntp_servers=127.24.249.148:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:18:09.735692 12054 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39175
I20260620 08:18:09.735807 12054 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:09.736017 12054 heartbeater.cc:511] Master 127.24.249.190:39175 requested a full tablet report, sending...
I20260620 08:18:09.736419 11734 ts_manager.cc:194] Registered new tserver with Master: d7af81d3ea53422ab7096cb89228daee (127.24.249.130:39257)
I20260620 08:18:09.736927 11734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.130:44501
W20260620 08:18:09.822760 12057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:09.822979 12057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:09.823002 12057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:09.824203 12057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:09.824249 12057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:09.824276 12057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:18:09.825676 12057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:09.825966 12057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:09.826282 12057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:09.832854 12065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.832872 12063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:09.832862 12062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:09.833156 12057 server_base.cc:1034] running on GCE node
I20260620 08:18:09.833417 12057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:09.833668 12057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:09.834882 12057 hybrid_clock.cc:648] HybridClock initialized: now 1781943489834844 us; error 55 us; skew 500 ppm
I20260620 08:18:09.836428 12057 webserver.cc:460] Webserver started at http://127.24.249.131:36955/ using document root <none> and password file <none>
I20260620 08:18:09.836769 12057 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:09.836850 12057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:09.837023 12057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:09.838286 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "adf970db67ae48b098d66588196028a2"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "ef55b7fbf618e85646c6c12866bef824"
server_key_iv: "88cd7488aca9be8fe5577545dbb7df2f"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.838691 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "adf970db67ae48b098d66588196028a2"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "ef55b7fbf618e85646c6c12866bef824"
server_key_iv: "88cd7488aca9be8fe5577545dbb7df2f"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.840154 12057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:09.841050 12071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.841261 12057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:09.841368 12057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "adf970db67ae48b098d66588196028a2"
format_stamp: "Formatted at 2026-06-20 08:18:09 on dist-test-slave-65w8"
server_key: "ef55b7fbf618e85646c6c12866bef824"
server_key_iv: "88cd7488aca9be8fe5577545dbb7df2f"
server_key_version: "encryptionkey@0"
I20260620 08:18:09.841475 12057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:09.855095 12057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:09.855425 12057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:09.855588 12057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:09.855831 12057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:09.856208 12057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:09.856271 12057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.856317 12057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:09.856349 12057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:09.863070 12057 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:39471
I20260620 08:18:09.863117 12184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:39471 every 8 connection(s)
I20260620 08:18:09.863507 12057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:18:09.867161 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 12057
I20260620 08:18:09.867264 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:18:09.867481 25574 external_mini_cluster.cc:1428] Setting key c57f9dd1dc32c27c6ceceb024c94d20e
I20260620 08:18:09.869647 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.864233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:46321 (local address 127.24.249.190:39175)
0620 08:18:09.864279 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.864281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.864538 (+ 257us) server_negotiation.cc:407] Connection header received
0620 08:18:09.865113 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.865114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.865126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.865196 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:09.865998 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.866534 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.867841 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.868125 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.868190 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.869254 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:09.869256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:09.869256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:09.869261 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:09.869278 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:09.869323 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:09.869409 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.869553 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.869584 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:18:09.869925 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.864209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:39175 (local address 127.24.249.131:46321)
0620 08:18:09.864403 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.864447 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.864994 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.865275 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.865279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.865287 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:09.865821 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.865825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.866626 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.866627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.867756 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.867760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.868224 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.868225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.868442 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.869080 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:09.869081 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:09.869112 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:09.869137 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:09.869427 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:09.869434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:09.869438 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.869522 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.869619 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260620 08:18:09.870360 12185 heartbeater.cc:347] Connected to a master server at 127.24.249.190:39175
I20260620 08:18:09.870473 12185 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:09.870669 12185 heartbeater.cc:511] Master 127.24.249.190:39175 requested a full tablet report, sending...
I20260620 08:18:09.871048 11734 ts_manager.cc:194] Registered new tserver with Master: adf970db67ae48b098d66588196028a2 (127.24.249.131:39471)
I20260620 08:18:09.871459 11734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.131:46321
I20260620 08:18:09.879784 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:18:09.884754 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.881259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48854 (local address 127.24.249.190:39175)
0620 08:18:09.881309 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.881311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.881447 (+ 136us) server_negotiation.cc:407] Connection header received
0620 08:18:09.881520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.881520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.881530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.881585 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:09.881779 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.882330 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.883256 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.883523 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.883616 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.884443 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:09.884445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:09.884445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:09.884450 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:09.884470 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:09.884475 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:09.884576 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.884683 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.884693 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:18:09.886627 11734 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:18:09.888252 11734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:18:09.894703 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.890539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:39257 (local address 127.0.0.1:47064)
0620 08:18:09.890887 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.890932 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.890995 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.891220 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.891224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.891226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.891306 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.891307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.892318 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.892318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.893355 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.893357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.894413 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.894414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.894496 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.894508 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.894575 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.894623 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":53,"threads_started":1}
I20260620 08:18:09.894799 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.890631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.24.249.130:39257)
0620 08:18:09.891014 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.891019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.891033 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:18:09.891080 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.891083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.891111 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.891202 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.891405 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.892242 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.893453 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.894212 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.894257 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.894351 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.894621 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.894669 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":93,"threads_started":1}
I20260620 08:18:09.895340 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.890735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:36479 (local address 127.0.0.1:39490)
0620 08:18:09.891043 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.891052 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.891093 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.891586 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.891587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.891589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.891625 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.891626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.892930 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.892931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.894005 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.894008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.895049 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.895050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.895108 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.895112 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.895157 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.895203 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":78,"threads_started":1}
I20260620 08:18:09.895413 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.890864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.24.249.129:36479)
0620 08:18:09.891408 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.891413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.891423 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:09.891471 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.891473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.891489 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.891574 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.891691 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.892861 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.894104 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.894861 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.894932 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.895013 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.895247 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.895290 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":87,"threads_started":1}
I20260620 08:18:09.895705 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.890645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.24.249.131:39471)
0620 08:18:09.890915 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.890921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.891076 (+ 155us) server_negotiation.cc:407] Connection header received
0620 08:18:09.891120 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.891123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.891161 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.891235 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.891468 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.892766 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.894061 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.894684 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.894763 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.894853 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.895308 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.895356 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":125,"threads_started":1}
I20260620 08:18:09.895987 11988 tablet_service.cc:1467] Processing CreateTablet for tablet a1f598765588436eb22aeec885a09adf (DEFAULT_TABLE table=test-table [id=f2c6d118e4884ad9bc709c087c6c0448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:09.896118 11857 tablet_service.cc:1467] Processing CreateTablet for tablet a1f598765588436eb22aeec885a09adf (DEFAULT_TABLE table=test-table [id=f2c6d118e4884ad9bc709c087c6c0448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:09.896275 11988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f598765588436eb22aeec885a09adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:09.896392 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f598765588436eb22aeec885a09adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:09.895208 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.890550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39471 (local address 127.0.0.1:56462)
0620 08:18:09.890887 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.890934 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.891008 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.891315 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.891316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.891319 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.891372 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.891374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.892872 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.892873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.893972 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.893975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.894895 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.894897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.894974 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.894978 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.895036 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.895101 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":90,"threads_started":1}
I20260620 08:18:09.899140 12199 tablet_bootstrap.cc:492] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Bootstrap starting.
I20260620 08:18:09.899142 12119 tablet_service.cc:1467] Processing CreateTablet for tablet a1f598765588436eb22aeec885a09adf (DEFAULT_TABLE table=test-table [id=f2c6d118e4884ad9bc709c087c6c0448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:09.899531 12119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f598765588436eb22aeec885a09adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:09.899665 12200 tablet_bootstrap.cc:492] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac: Bootstrap starting.
I20260620 08:18:09.899812 12199 tablet_bootstrap.cc:654] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:09.900044 12199 log.cc:824] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:09.900358 12200 tablet_bootstrap.cc:654] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:09.900694 12199 tablet_bootstrap.cc:492] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: No bootstrap required, opened a new log
I20260620 08:18:09.900688 12200 log.cc:824] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:09.900739 12199 ts_tablet_manager.cc:1397] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:09.901635 12200 tablet_bootstrap.cc:492] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac: No bootstrap required, opened a new log
I20260620 08:18:09.901700 12200 ts_tablet_manager.cc:1397] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:09.903013 12199 raft_consensus.cc:348] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.903260 12199 raft_consensus.cc:374] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:09.903308 12199 raft_consensus.cc:729] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7af81d3ea53422ab7096cb89228daee, State: Initialized, Role: FOLLOWER
I20260620 08:18:09.903427 12199 consensus_queue.cc:260] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.903697 12199 ts_tablet_manager.cc:1428] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:09.903716 12203 tablet_bootstrap.cc:492] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Bootstrap starting.
I20260620 08:18:09.903815 12054 heartbeater.cc:503] Master 127.24.249.190:39175 was elected leader, sending a full tablet report...
I20260620 08:18:09.904410 12203 tablet_bootstrap.cc:654] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:09.904753 12203 log.cc:824] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:09.904667 12200 raft_consensus.cc:348] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.904843 12200 raft_consensus.cc:374] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:09.904879 12200 raft_consensus.cc:729] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95fe2a777104113bb95a3fd8a4113ac, State: Initialized, Role: FOLLOWER
I20260620 08:18:09.905005 12200 consensus_queue.cc:260] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.905290 12200 ts_tablet_manager.cc:1428] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 08:18:09.905452 11923 heartbeater.cc:503] Master 127.24.249.190:39175 was elected leader, sending a full tablet report...
I20260620 08:18:09.905620 12203 tablet_bootstrap.cc:492] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: No bootstrap required, opened a new log
I20260620 08:18:09.905715 12203 ts_tablet_manager.cc:1397] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:09.908064 12203 raft_consensus.cc:348] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.908202 12203 raft_consensus.cc:374] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:09.908227 12203 raft_consensus.cc:729] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf970db67ae48b098d66588196028a2, State: Initialized, Role: FOLLOWER
I20260620 08:18:09.908357 12203 consensus_queue.cc:260] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.908659 12203 ts_tablet_manager.cc:1428] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:09.908740 12185 heartbeater.cc:503] Master 127.24.249.190:39175 was elected leader, sending a full tablet report...
I20260620 08:18:09.921382 12206 raft_consensus.cc:482] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:18:09.921516 12206 raft_consensus.cc:504] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.922046 12206 leader_election.cc:288] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adf970db67ae48b098d66588196028a2 (127.24.249.131:39471), d7af81d3ea53422ab7096cb89228daee (127.24.249.130:39257)
I20260620 08:18:09.925261 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.922212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39471 (local address 127.24.249.129:38017)
0620 08:18:09.922284 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.922296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.922390 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.922563 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.922564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.922565 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.922644 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.922646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.923413 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.923414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.924347 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.924350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.925012 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.925014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.925095 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.925099 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.925155 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.925199 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":23}
I20260620 08:18:09.925319 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.922272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:38017 (local address 127.24.249.131:39471)
0620 08:18:09.922336 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.922339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.922405 (+ 66us) server_negotiation.cc:407] Connection header received
0620 08:18:09.922474 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.922475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.922487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.922551 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.922734 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.923292 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.924444 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.924922 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.924982 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.924989 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.925172 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.925218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:18:09.925437 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.922241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:39257 (local address 127.24.249.129:46605)
0620 08:18:09.922416 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.922429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:09.922495 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:09.922641 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:09.922641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:09.922642 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:09.922734 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.922736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.923558 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.923560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.924522 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:09.924526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.925120 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.925121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:09.925238 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.925243 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:18:09.925307 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 08:18:09.925351 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260620 08:18:09.925501 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.922346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:46605 (local address 127.24.249.130:39257)
0620 08:18:09.922390 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.922393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.922503 (+ 110us) server_negotiation.cc:407] Connection header received
0620 08:18:09.922566 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.922567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.922578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.922630 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:09.922838 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.923459 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.924623 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.925031 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.925112 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.925120 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.925324 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.925376 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:18:09.925752 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f598765588436eb22aeec885a09adf" candidate_uuid: "a95fe2a777104113bb95a3fd8a4113ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf970db67ae48b098d66588196028a2" is_pre_election: true
I20260620 08:18:09.925805 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f598765588436eb22aeec885a09adf" candidate_uuid: "a95fe2a777104113bb95a3fd8a4113ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7af81d3ea53422ab7096cb89228daee" is_pre_election: true
I20260620 08:18:09.925972 12008 raft_consensus.cc:2454] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a95fe2a777104113bb95a3fd8a4113ac in term 0.
I20260620 08:18:09.925973 12139 raft_consensus.cc:2454] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a95fe2a777104113bb95a3fd8a4113ac in term 0.
I20260620 08:18:09.926213 11813 leader_election.cc:302] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a95fe2a777104113bb95a3fd8a4113ac, d7af81d3ea53422ab7096cb89228daee; no voters:
I20260620 08:18:09.926373 12206 raft_consensus.cc:2781] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:18:09.926440 12206 raft_consensus.cc:482] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:18:09.926486 12206 raft_consensus.cc:3037] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:09.927242 12206 raft_consensus.cc:504] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.927446 12206 leader_election.cc:288] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [CANDIDATE]: Term 1 election: Requested vote from peers adf970db67ae48b098d66588196028a2 (127.24.249.131:39471), d7af81d3ea53422ab7096cb89228daee (127.24.249.130:39257)
I20260620 08:18:09.927594 12139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f598765588436eb22aeec885a09adf" candidate_uuid: "a95fe2a777104113bb95a3fd8a4113ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf970db67ae48b098d66588196028a2"
I20260620 08:18:09.927615 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f598765588436eb22aeec885a09adf" candidate_uuid: "a95fe2a777104113bb95a3fd8a4113ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7af81d3ea53422ab7096cb89228daee"
I20260620 08:18:09.927677 12139 raft_consensus.cc:3037] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:09.927685 12008 raft_consensus.cc:3037] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:09.928619 12008 raft_consensus.cc:2454] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95fe2a777104113bb95a3fd8a4113ac in term 1.
I20260620 08:18:09.928664 12139 raft_consensus.cc:2454] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95fe2a777104113bb95a3fd8a4113ac in term 1.
I20260620 08:18:09.928861 11812 leader_election.cc:302] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a95fe2a777104113bb95a3fd8a4113ac, adf970db67ae48b098d66588196028a2; no voters:
I20260620 08:18:09.929013 12206 raft_consensus.cc:2781] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:09.929204 12206 raft_consensus.cc:686] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 1 LEADER]: Becoming Leader. State: Replica: a95fe2a777104113bb95a3fd8a4113ac, State: Running, Role: LEADER
I20260620 08:18:09.929340 12206 consensus_queue.cc:237] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } }
I20260620 08:18:09.930413 11733 catalog_manager.cc:5495] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a95fe2a777104113bb95a3fd8a4113ac (127.24.249.129). New cstate: current_term: 1 leader_uuid: "a95fe2a777104113bb95a3fd8a4113ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a95fe2a777104113bb95a3fd8a4113ac" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 36479 } health_report { overall_health: HEALTHY } } }
I20260620 08:18:09.942055 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.939619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.24.249.129:36479)
0620 08:18:09.939674 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:09.939676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:09.939685 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:09.939726 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:09.939727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:09.939747 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:09.939834 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:09.939957 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.940431 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.941191 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:09.941414 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:09.941502 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:09.941642 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:09.941739 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:09.941849 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:09.941939 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 08:18:09.941949 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:18:09.944082 12008 raft_consensus.cc:1261] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 1 FOLLOWER]: Refusing update from remote peer a95fe2a777104113bb95a3fd8a4113ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:09.944082 12139 raft_consensus.cc:1261] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 1 FOLLOWER]: Refusing update from remote peer a95fe2a777104113bb95a3fd8a4113ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:09.944514 12209 consensus_queue.cc:1035] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf970db67ae48b098d66588196028a2" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:09.944664 12206 consensus_queue.cc:1035] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7af81d3ea53422ab7096cb89228daee" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 39257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:09.950012 11734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:09.950101 11734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:09.950855 11734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d511fa8180140f29e5330a453812e30: Sending DeleteTablet for 3 replicas of tablet a1f598765588436eb22aeec885a09adf
I20260620 08:18:09.951201 12119 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f598765588436eb22aeec885a09adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:09 UTC) from {username='slave'} at 127.0.0.1:56462
I20260620 08:18:09.951295 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f598765588436eb22aeec885a09adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:09 UTC) from {username='slave'} at 127.0.0.1:39490
I20260620 08:18:09.951300 11988 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f598765588436eb22aeec885a09adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:09 UTC) from {username='slave'} at 127.0.0.1:47064
I20260620 08:18:09.951511 12222 tablet_replica.cc:330] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: stopping tablet replica
I20260620 08:18:09.951512 12223 tablet_replica.cc:330] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac: stopping tablet replica
I20260620 08:18:09.951610 12222 raft_consensus.cc:2229] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:09.951610 12223 raft_consensus.cc:2229] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:18:09.951702 12223 raft_consensus.cc:2258] T a1f598765588436eb22aeec885a09adf P a95fe2a777104113bb95a3fd8a4113ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:09.951709 12222 raft_consensus.cc:2258] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:09.951870 12224 tablet_replica.cc:330] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: stopping tablet replica
I20260620 08:18:09.951968 12224 raft_consensus.cc:2229] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:09.952061 12224 raft_consensus.cc:2258] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:09.952355 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 11795
I20260620 08:18:09.953095 12224 ts_tablet_manager.cc:1905] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:09.955018 12224 ts_tablet_manager.cc:1918] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:09.955083 12224 log.cc:1196] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/a1f598765588436eb22aeec885a09adf
I20260620 08:18:09.955324 12224 ts_tablet_manager.cc:1939] T a1f598765588436eb22aeec885a09adf P d7af81d3ea53422ab7096cb89228daee: Deleting consensus metadata
I20260620 08:18:09.955864 11721 catalog_manager.cc:4841] TS d7af81d3ea53422ab7096cb89228daee (127.24.249.130:39257): tablet a1f598765588436eb22aeec885a09adf (table test-table [id=f2c6d118e4884ad9bc709c087c6c0448]) successfully deleted
I20260620 08:18:09.958190 12222 ts_tablet_manager.cc:1905] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 08:18:09.958441 11721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:36479 (error 108)
I20260620 08:18:09.958779 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 11926
I20260620 08:18:09.959779 12222 ts_tablet_manager.cc:1918] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:09.959877 12222 log.cc:1196] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/a1f598765588436eb22aeec885a09adf
I20260620 08:18:09.960124 12222 ts_tablet_manager.cc:1939] T a1f598765588436eb22aeec885a09adf P adf970db67ae48b098d66588196028a2: Deleting consensus metadata
I20260620 08:18:09.960564 11720 catalog_manager.cc:4841] TS adf970db67ae48b098d66588196028a2 (127.24.249.131:39471): tablet a1f598765588436eb22aeec885a09adf (table test-table [id=f2c6d118e4884ad9bc709c087c6c0448]) successfully deleted
I20260620 08:18:09.960662 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:09.960452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:36479 (local address 127.0.0.1:39510)
0620 08:18:09.960510 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:09.960558 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 08:18:09.960772 11721 catalog_manager.cc:4584] TS a95fe2a777104113bb95a3fd8a4113ac (127.24.249.129:36479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1f598765588436eb22aeec885a09adf: Network error: Client connection negotiation failed: client connection to 127.24.249.129:36479: connect: Connection refused (error 111)
I20260620 08:18:09.964605 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 12057
I20260620 08:18:09.970263 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 11703
2026-06-20T08:18:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (711 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:10 dist-test-slave-65w8 krb5kdc[12228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:10 dist-test-slave-65w8 krb5kdc[12228](info): set up 2 sockets
Jun 20 08:18:10 dist-test-slave-65w8 krb5kdc[12228](info): commencing operation
krb5kdc: starting...
W20260620 08:18:11.074038 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:11.202184 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:33049
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:33049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:11.284482 12241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:11.284687 12241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:11.284709 12241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:11.285931 12241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:11.285981 12241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:11.285995 12241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:11.286008 12241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:11.286020 12241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:11.287613 12241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:33049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:33049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:11.287875 12241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:11.288142 12241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:11.293085 12249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:11.293154 12247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:11.293095 12246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:11.293501 12241 server_base.cc:1034] running on GCE node
I20260620 08:18:11.293782 12241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:11.294103 12241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:11.295275 12241 hybrid_clock.cc:648] HybridClock initialized: now 1781943491295256 us; error 53 us; skew 500 ppm
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:11.297955 12241 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:11.298534 12241 webserver.cc:460] Webserver started at http://127.24.249.190:36241/ using document root <none> and password file <none>
I20260620 08:18:11.298803 12241 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:11.298875 12241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:11.299028 12241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:11.300113 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "c97bd37ae00c441fa9915d5478aaf241"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "9ee8dd0cb18019b9ead6cc64fd69a353"
server_key_iv: "9f79df15e10b82bf3f8139c47f1387a1"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.300491 12241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "c97bd37ae00c441fa9915d5478aaf241"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "9ee8dd0cb18019b9ead6cc64fd69a353"
server_key_iv: "9f79df15e10b82bf3f8139c47f1387a1"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.301954 12241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:11.302755 12256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.302995 12241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:11.303123 12241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "c97bd37ae00c441fa9915d5478aaf241"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "9ee8dd0cb18019b9ead6cc64fd69a353"
server_key_iv: "9f79df15e10b82bf3f8139c47f1387a1"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.303217 12241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:11.323256 12241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:11.325309 12241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:11.325484 12241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:11.329692 12241 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:33049
I20260620 08:18:11.329730 12308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:33049 every 8 connection(s)
I20260620 08:18:11.330142 12241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:11.331063 12309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:11.333581 12309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241: Bootstrap starting.
I20260620 08:18:11.334461 12309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:11.334853 12309 log.cc:824] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:11.335866 12309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241: No bootstrap required, opened a new log
I20260620 08:18:11.337558 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 12241
I20260620 08:18:11.337684 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:11.337915 25574 external_mini_cluster.cc:1428] Setting key b4c2f7269baa3393c0fce64ed7438979
I20260620 08:18:11.338330 12309 raft_consensus.cc:348] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97bd37ae00c441fa9915d5478aaf241" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33049 } }
I20260620 08:18:11.338543 12309 raft_consensus.cc:374] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:11.338595 12309 raft_consensus.cc:729] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97bd37ae00c441fa9915d5478aaf241, State: Initialized, Role: FOLLOWER
I20260620 08:18:11.338747 12309 consensus_queue.cc:260] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97bd37ae00c441fa9915d5478aaf241" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33049 } }
I20260620 08:18:11.338863 12309 raft_consensus.cc:388] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:11.338914 12309 raft_consensus.cc:482] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:11.338972 12309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:11.339967 12309 raft_consensus.cc:504] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97bd37ae00c441fa9915d5478aaf241" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33049 } }
I20260620 08:18:11.340106 12309 leader_election.cc:302] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c97bd37ae00c441fa9915d5478aaf241; no voters:
I20260620 08:18:11.340329 12309 leader_election.cc:288] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:11.340365 12314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:11.340481 12314 raft_consensus.cc:686] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [term 1 LEADER]: Becoming Leader. State: Replica: c97bd37ae00c441fa9915d5478aaf241, State: Running, Role: LEADER
I20260620 08:18:11.340574 12314 consensus_queue.cc:237] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97bd37ae00c441fa9915d5478aaf241" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33049 } }
I20260620 08:18:11.340631 12309 sys_catalog.cc:564] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:11.340979 12314 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c97bd37ae00c441fa9915d5478aaf241. Latest consensus state: current_term: 1 leader_uuid: "c97bd37ae00c441fa9915d5478aaf241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97bd37ae00c441fa9915d5478aaf241" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33049 } } }
I20260620 08:18:11.341087 12314 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:11.341355 12320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:11.341928 12320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:11.340991 12315 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c97bd37ae00c441fa9915d5478aaf241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97bd37ae00c441fa9915d5478aaf241" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 33049 } } }
I20260620 08:18:11.342998 12315 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241 [sys.catalog]: This master's current role is: LEADER
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:11.346578 12320 catalog_manager.cc:1347] Generated new cluster ID: 708d6a880a634454b6af83302a4ca0f5
I20260620 08:18:11.347466 12320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:11.349278 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.338459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.24.249.190:33049)
0620 08:18:11.338664 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.338673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.338704 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:18:11.338839 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.338845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.338914 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.339055 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:11.339700 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.340962 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.342036 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.342520 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.342650 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.345377 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:11.345380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:11.345381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:11.345392 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:11.347347 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.347794 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.347798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.347799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.347954 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.348190 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.348191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.348192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.348548 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:11.348735 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.348850 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.349055 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260620 08:18:11.357954 12320 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:11.358749 12320 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:11.362129 12320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241: Generated new TSK 0
I20260620 08:18:11.362350 12320 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.249.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
Entry for principal kudu/127.24.249.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab.
I20260620 08:18:11.398866 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:33049
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:11.483160 12335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:11.483428 12335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:11.483453 12335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:11.484762 12335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:11.484809 12335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:11.484838 12335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:18:11.486372 12335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.129.keytab
--principal=kudu/127.24.249.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:33049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:11.486629 12335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:11.486891 12335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:11.492323 12343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:11.492422 12341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:11.492324 12340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:11.492539 12335 server_base.cc:1034] running on GCE node
I20260620 08:18:11.492799 12335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:11.493058 12335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:11.494226 12335 hybrid_clock.cc:648] HybridClock initialized: now 1781943491494221 us; error 37 us; skew 500 ppm
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:11.496887 12335 init.cc:377] Logged in from keytab as kudu/127.24.249.129@KRBTEST.COM (short username kudu)
I20260620 08:18:11.497335 12335 webserver.cc:460] Webserver started at http://127.24.249.129:33179/ using document root <none> and password file <none>
I20260620 08:18:11.497565 12335 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:11.497632 12335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:11.497783 12335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:11.499013 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "8a2fe31bca4040d2aec5437edfe48a59"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "6391acef5bbeba3ac013aa7457c97c2d"
server_key_iv: "8ae195babe0e7a33973fc05199bba781"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.499367 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "8a2fe31bca4040d2aec5437edfe48a59"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "6391acef5bbeba3ac013aa7457c97c2d"
server_key_iv: "8ae195babe0e7a33973fc05199bba781"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.500706 12335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:11.501484 12350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.501787 12335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:11.501924 12335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "8a2fe31bca4040d2aec5437edfe48a59"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "6391acef5bbeba3ac013aa7457c97c2d"
server_key_iv: "8ae195babe0e7a33973fc05199bba781"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.502019 12335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:11.512769 12335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:11.515192 12335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:11.515363 12335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:11.515609 12335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:11.515941 12335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:11.515997 12335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.516037 12335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:11.516067 12335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.522495 12335 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:34801
I20260620 08:18:11.522545 12463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:34801 every 8 connection(s)
I20260620 08:18:11.522854 12335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:18:11.524636 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 12335
I20260620 08:18:11.524731 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:18:11.524948 25574 external_mini_cluster.cc:1428] Setting key 49bb86c571949010ea39805e7de35607
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.129@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.130@KRBTEST.COM; defaulting to no policy
I20260620 08:18:11.533331 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.523542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:33525 (local address 127.24.249.190:33049)
0620 08:18:11.523627 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.523629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.523835 (+ 206us) server_negotiation.cc:407] Connection header received
0620 08:18:11.524348 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.524350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.524365 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.524415 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:11.525662 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.526489 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.527752 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.527965 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.528021 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.530277 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:11.530279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:11.530279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:11.530287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:11.531825 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.532293 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.532294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.532294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.532363 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.532624 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.532625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.532625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.532748 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:11.532827 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.533117 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.533233 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 08:18:11.533530 12466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.523498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:33049 (local address 127.24.249.129:33525)
0620 08:18:11.523679 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:11.523715 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:11.524246 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:11.524495 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:11.524498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:11.525279 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:11.525568 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.525574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.526579 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.526579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.527661 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.527663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.528050 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.528051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.528210 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.528803 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:11.528812 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:11.530180 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:11.531946 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:11.531954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:11.531956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:11.532201 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:11.532442 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:11.532442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:11.532443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:11.532554 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:11.532818 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:11.532825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:11.532978 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 08:18:11.533083 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:18:11.533247 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260620 08:18:11.534190 12464 heartbeater.cc:347] Connected to a master server at 127.24.249.190:33049
I20260620 08:18:11.534337 12464 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:11.534580 12464 heartbeater.cc:511] Master 127.24.249.190:33049 requested a full tablet report, sending...
I20260620 08:18:11.535090 12273 ts_manager.cc:194] Registered new tserver with Master: 8a2fe31bca4040d2aec5437edfe48a59 (127.24.249.129:34801)
I20260620 08:18:11.535717 12273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.129@KRBTEST.COM'} at 127.24.249.129:33525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
Entry for principal kudu/127.24.249.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab.
I20260620 08:18:11.553381 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:33049
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:11.635735 12469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:11.635988 12469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:11.636027 12469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:11.637193 12469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:11.637275 12469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:11.637322 12469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:18:11.638850 12469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.130.keytab
--principal=kudu/127.24.249.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:33049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:11.639119 12469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:11.639364 12469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:11.644933 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:11.645072 12469 server_base.cc:1034] running on GCE node
W20260620 08:18:11.644933 12474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:11.645078 12477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:11.645478 12469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:11.645744 12469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:11.646953 12469 hybrid_clock.cc:648] HybridClock initialized: now 1781943491646938 us; error 38 us; skew 500 ppm
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:11.650732 12469 init.cc:377] Logged in from keytab as kudu/127.24.249.130@KRBTEST.COM (short username kudu)
I20260620 08:18:11.651527 12469 webserver.cc:460] Webserver started at http://127.24.249.130:46849/ using document root <none> and password file <none>
I20260620 08:18:11.651916 12469 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:11.652005 12469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:11.652170 12469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:11.653376 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "ba33a6c27967413fa2fb023978e615a2"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "70c07cb8f9360df39725e1fc9c2e328d"
server_key_iv: "0fba43916808d36af15a2c5fd8cb2f92"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.653805 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "ba33a6c27967413fa2fb023978e615a2"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "70c07cb8f9360df39725e1fc9c2e328d"
server_key_iv: "0fba43916808d36af15a2c5fd8cb2f92"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.655601 12469 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:11.656566 12484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.656889 12469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:11.657018 12469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "ba33a6c27967413fa2fb023978e615a2"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "70c07cb8f9360df39725e1fc9c2e328d"
server_key_iv: "0fba43916808d36af15a2c5fd8cb2f92"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.657104 12469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:11.679855 12469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:11.682250 12469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:11.682482 12469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:11.682753 12469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:11.683089 12469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:11.683145 12469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.683188 12469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:11.683216 12469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.689832 12469 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:44983
I20260620 08:18:11.689926 12597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:44983 every 8 connection(s)
I20260620 08:18:11.690245 12469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.130@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:11.699404 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 12469
I20260620 08:18:11.699546 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:18:11.699728 25574 external_mini_cluster.cc:1428] Setting key 5aea5692d31c27d9bd0fcbd6b60418a7
I20260620 08:18:11.700279 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.691021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:44565 (local address 127.24.249.190:33049)
0620 08:18:11.691066 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.691068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.691294 (+ 226us) server_negotiation.cc:407] Connection header received
0620 08:18:11.691849 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.691850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.691863 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.691922 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:11.692917 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.693436 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.694553 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.694762 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.694853 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.697056 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:11.697058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:11.697058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:11.697066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:11.698628 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.699197 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.699198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.699198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.699263 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.699510 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.699511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.699512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.699667 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:11.699751 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.700099 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.700188 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:18:11.700495 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.691005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:33049 (local address 127.24.249.130:44565)
0620 08:18:11.691165 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:11.691203 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:11.691736 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:11.691999 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:11.692003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:11.692590 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:11.692793 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.692798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.693539 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.693540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.694451 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.694454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.694869 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.694870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.694999 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.695494 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:11.695502 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:11.696953 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:11.698755 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:11.698763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:11.698765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:11.699098 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:11.699340 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:11.699341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:11.699341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:11.699440 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:11.699740 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:11.699747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:11.699960 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 08:18:11.700075 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 08:18:11.700246 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260620 08:18:11.701017 12598 heartbeater.cc:347] Connected to a master server at 127.24.249.190:33049
I20260620 08:18:11.701133 12598 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:11.701313 12598 heartbeater.cc:511] Master 127.24.249.190:33049 requested a full tablet report, sending...
I20260620 08:18:11.701771 12273 ts_manager.cc:194] Registered new tserver with Master: ba33a6c27967413fa2fb023978e615a2 (127.24.249.130:44983)
I20260620 08:18:11.702209 12273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.130@KRBTEST.COM'} at 127.24.249.130:44565
WARNING: no policy specified for kudu/127.24.249.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
Entry for principal kudu/127.24.249.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab.
I20260620 08:18:11.729357 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:33049
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:11.818185 12603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:11.818470 12603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:11.818513 12603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:11.819761 12603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:11.819839 12603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:11.819882 12603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:18:11.821476 12603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.131.keytab
--principal=kudu/127.24.249.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:33049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:11.821766 12603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:11.822038 12603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:11.827445 12609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:11.827589 12603 server_base.cc:1034] running on GCE node
W20260620 08:18:11.827374 12611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:11.827365 12608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:11.827888 12603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:11.828104 12603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:11.829314 12603 hybrid_clock.cc:648] HybridClock initialized: now 1781943491829280 us; error 46 us; skew 500 ppm
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:11.832382 12603 init.cc:377] Logged in from keytab as kudu/127.24.249.131@KRBTEST.COM (short username kudu)
I20260620 08:18:11.832913 12603 webserver.cc:460] Webserver started at http://127.24.249.131:40025/ using document root <none> and password file <none>
I20260620 08:18:11.833184 12603 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:11.833280 12603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:11.833518 12603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:11.834774 12603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "b0100bbfa557482cac53fa6a752d09c0"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "a6c2dc7b4cd620238b9529d2b606b359"
server_key_iv: "fc0ea2d61ba7031dc98f92aab01a3de5"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.835153 12603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "b0100bbfa557482cac53fa6a752d09c0"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "a6c2dc7b4cd620238b9529d2b606b359"
server_key_iv: "fc0ea2d61ba7031dc98f92aab01a3de5"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.836648 12603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:18:11.837499 12618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.837697 12603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:11.837801 12603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "b0100bbfa557482cac53fa6a752d09c0"
format_stamp: "Formatted at 2026-06-20 08:18:11 on dist-test-slave-65w8"
server_key: "a6c2dc7b4cd620238b9529d2b606b359"
server_key_iv: "fc0ea2d61ba7031dc98f92aab01a3de5"
server_key_version: "encryptionkey@0"
I20260620 08:18:11.837922 12603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:11.862128 12603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:11.864337 12603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:11.864502 12603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:11.864753 12603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:11.865129 12603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:11.865183 12603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.865226 12603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:11.865252 12603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:11.871313 12603 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:39921
I20260620 08:18:11.871398 12731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:39921 every 8 connection(s)
I20260620 08:18:11.871732 12603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:18:11.875304 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 12603
I20260620 08:18:11.875406 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:18:11.875595 25574 external_mini_cluster.cc:1428] Setting key 8ce8f65166fc0a09a1bf03f89c2c9973
Jun 20 08:18:11 dist-test-slave-65w8 krb5kdc[12228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943491, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.131@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:11.882100 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.872557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:38289 (local address 127.24.249.190:33049)
0620 08:18:11.872608 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.872610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.872779 (+ 169us) server_negotiation.cc:407] Connection header received
0620 08:18:11.873284 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.873285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.873297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.873373 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:11.874424 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.874934 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.876538 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.876756 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.876813 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.879151 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:11.879153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:11.879153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:11.879161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:11.880630 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.881075 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.881076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.881076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.881136 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.881391 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.881392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.881392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.881513 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:11.881585 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.881914 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.881998 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:18:11.882313 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.872479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:33049 (local address 127.24.249.131:38289)
0620 08:18:11.872642 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:11.872685 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:11.873158 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:11.873429 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:11.873432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:11.874087 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:11.874296 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.874302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.875057 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.875058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.876441 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.876451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.876832 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.876833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.876940 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.877472 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:11.877481 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:11.879046 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:11.880735 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:11.880743 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:11.880746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:11.880987 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:11.881202 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:11.881203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:11.881203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:11.881320 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:11.881588 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:11.881596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:11.881739 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 08:18:11.881895 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 08:18:11.882071 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260620 08:18:11.882820 12732 heartbeater.cc:347] Connected to a master server at 127.24.249.190:33049
I20260620 08:18:11.882915 12732 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:11.883101 12732 heartbeater.cc:511] Master 127.24.249.190:33049 requested a full tablet report, sending...
I20260620 08:18:11.883653 12273 ts_manager.cc:194] Registered new tserver with Master: b0100bbfa557482cac53fa6a752d09c0 (127.24.249.131:39921)
I20260620 08:18:11.884063 12273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.249.131@KRBTEST.COM'} at 127.24.249.131:38289
I20260620 08:18:11.887660 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:18:11.895660 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.889066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.24.249.190:33049)
0620 08:18:11.889111 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.889113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.889282 (+ 169us) server_negotiation.cc:407] Connection header received
0620 08:18:11.889325 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.889326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.889337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.889399 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:11.890145 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.890621 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.891318 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.891531 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.891586 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.892757 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:11.892758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:11.892758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:11.892764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:11.894323 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.894720 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.894721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.894721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.894771 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:11.894991 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:11.894992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:11.894993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:11.895204 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:11.895309 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.895460 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.895548 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:18:11.897289 12273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:18:11.898988 12273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:18:11.905630 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.901576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.24.249.130:44983)
0620 08:18:11.901730 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.901735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.901909 (+ 174us) server_negotiation.cc:407] Connection header received
0620 08:18:11.902024 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.902027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.902043 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.902112 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:11.902345 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.903052 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.904466 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.905189 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.905267 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.905321 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.905496 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.905541 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260620 08:18:11.905629 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.901314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:44983 (local address 127.0.0.1:34222)
0620 08:18:11.901817 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:11.901828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:11.901940 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:11.902192 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:11.902193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:11.902195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:11.902266 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.902268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.903156 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.903157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.904360 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.904362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.905277 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.905278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.905376 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.905392 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 08:18:11.905473 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 08:18:11.905538 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":23,"thread_start_us":112,"threads_started":1}
I20260620 08:18:11.906098 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.901314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39921 (local address 127.0.0.1:49230)
0620 08:18:11.901476 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:11.901517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:11.901582 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:11.902115 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:11.902118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:11.902121 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:11.902197 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.902199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.903594 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.903595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.904651 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.904653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.905854 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.905855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.905947 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.905951 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:11.905986 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 08:18:11.906027 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":48,"threads_started":1}
I20260620 08:18:11.906548 12532 tablet_service.cc:1467] Processing CreateTablet for tablet 950beceab9204a488133467becb9f2af (DEFAULT_TABLE table=test-table [id=8af3abd4ecf64bacb625d9bad144b366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:11.906855 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.901571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.24.249.131:39921)
0620 08:18:11.901785 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.901788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.901798 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:11.901842 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.901843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.901911 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.902280 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:11.902351 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.903484 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.904722 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.905601 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.906309 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.906380 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.906435 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.906773 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260620 08:18:11.906924 12532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950beceab9204a488133467becb9f2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:11.907135 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.901314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:34801 (local address 127.0.0.1:49684)
0620 08:18:11.901697 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:11.901705 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:11.901744 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:11.902195 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:11.902196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:11.902198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:11.902275 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.902276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.904771 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.904772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.905884 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:11.905887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.906902 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.906904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:11.906968 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.906971 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:11.907025 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 08:18:11.907065 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":29,"thread_start_us":118,"threads_started":1}
I20260620 08:18:11.907169 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:11.901755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.24.249.129:34801)
0620 08:18:11.901982 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:11.901986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:11.901995 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:11.902034 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:11.902036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:11.902052 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:11.902121 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:11.902354 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.903472 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.905970 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:11.906723 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:11.906787 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:11.906833 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:11.907045 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 08:18:11.907094 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":132,"threads_started":1}
I20260620 08:18:11.907861 12398 tablet_service.cc:1467] Processing CreateTablet for tablet 950beceab9204a488133467becb9f2af (DEFAULT_TABLE table=test-table [id=8af3abd4ecf64bacb625d9bad144b366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:11.908205 12398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950beceab9204a488133467becb9f2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:11.910502 12749 tablet_bootstrap.cc:492] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Bootstrap starting.
I20260620 08:18:11.910482 12666 tablet_service.cc:1467] Processing CreateTablet for tablet 950beceab9204a488133467becb9f2af (DEFAULT_TABLE table=test-table [id=8af3abd4ecf64bacb625d9bad144b366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:11.910825 12666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950beceab9204a488133467becb9f2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:11.911206 12749 tablet_bootstrap.cc:654] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:11.911501 12750 tablet_bootstrap.cc:492] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: Bootstrap starting.
I20260620 08:18:11.911769 12749 log.cc:824] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:11.912288 12750 tablet_bootstrap.cc:654] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:11.912623 12749 tablet_bootstrap.cc:492] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: No bootstrap required, opened a new log
I20260620 08:18:11.912784 12750 log.cc:824] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:11.912845 12749 ts_tablet_manager.cc:1397] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:18:11.913460 12750 tablet_bootstrap.cc:492] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: No bootstrap required, opened a new log
I20260620 08:18:11.913542 12750 ts_tablet_manager.cc:1397] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:11.914287 12753 tablet_bootstrap.cc:492] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Bootstrap starting.
I20260620 08:18:11.914995 12753 tablet_bootstrap.cc:654] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:11.914943 12749 raft_consensus.cc:348] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:11.915126 12749 raft_consensus.cc:374] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:11.915166 12749 raft_consensus.cc:729] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba33a6c27967413fa2fb023978e615a2, State: Initialized, Role: FOLLOWER
I20260620 08:18:11.915282 12749 consensus_queue.cc:260] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:11.915572 12753 log.cc:824] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:11.915591 12749 ts_tablet_manager.cc:1428] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:11.915575 12750 raft_consensus.cc:348] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:11.915751 12750 raft_consensus.cc:374] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:11.915771 12598 heartbeater.cc:503] Master 127.24.249.190:33049 was elected leader, sending a full tablet report...
I20260620 08:18:11.915802 12750 raft_consensus.cc:729] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2fe31bca4040d2aec5437edfe48a59, State: Initialized, Role: FOLLOWER
I20260620 08:18:11.915925 12750 consensus_queue.cc:260] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:11.916227 12750 ts_tablet_manager.cc:1428] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:11.916251 12464 heartbeater.cc:503] Master 127.24.249.190:33049 was elected leader, sending a full tablet report...
I20260620 08:18:11.916853 12753 tablet_bootstrap.cc:492] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: No bootstrap required, opened a new log
I20260620 08:18:11.916915 12753 ts_tablet_manager.cc:1397] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 08:18:11.918432 12753 raft_consensus.cc:348] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:11.918558 12753 raft_consensus.cc:374] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:11.918579 12753 raft_consensus.cc:729] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0100bbfa557482cac53fa6a752d09c0, State: Initialized, Role: FOLLOWER
I20260620 08:18:11.918684 12753 consensus_queue.cc:260] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:11.918908 12753 ts_tablet_manager.cc:1428] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 08:18:11.918970 12732 heartbeater.cc:503] Master 127.24.249.190:33049 was elected leader, sending a full tablet report...
W20260620 08:18:11.940824 12599 tablet.cc:2369] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 08:18:12.023658 12465 tablet.cc:2369] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:18:12.071276 12754 raft_consensus.cc:482] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:18:12.071468 12754 raft_consensus.cc:504] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:12.071962 12754 leader_election.cc:288] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a2fe31bca4040d2aec5437edfe48a59 (127.24.249.129:34801), b0100bbfa557482cac53fa6a752d09c0 (127.24.249.131:39921)
I20260620 08:18:12.075094 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.072205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:39921 (local address 127.24.249.130:52463)
0620 08:18:12.072276 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.072290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.072411 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.072598 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.072599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.072601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:12.072680 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.072682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.073358 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.073359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.074235 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.074237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.074805 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.074806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.074897 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.074901 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.074969 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.075010 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":4}
I20260620 08:18:12.075088 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.072322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:52463 (local address 127.24.249.131:39921)
0620 08:18:12.072370 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.072372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.072382 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:12.072505 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.072506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.072517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.072594 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:12.072752 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.073269 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.074336 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.074713 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.074803 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.074812 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.074974 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.075014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 08:18:12.075096 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.072316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:59111 (local address 127.24.249.129:34801)
0620 08:18:12.072377 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.072380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.072534 (+ 154us) server_negotiation.cc:407] Connection header received
0620 08:18:12.072606 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.072607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.072619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.072693 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:12.072880 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.073367 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.074352 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.074720 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.074806 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.074814 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.074988 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.075029 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 08:18:12.075124 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.072205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:34801 (local address 127.24.249.130:59111)
0620 08:18:12.072451 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.072459 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.072544 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.072707 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.072708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.072710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:12.072789 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.072790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.073458 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.073459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.074247 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.074249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.074806 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.074807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.074906 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.074910 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.074981 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.075021 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":84,"thread_start_us":99,"threads_started":1}
I20260620 08:18:12.075441 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950beceab9204a488133467becb9f2af" candidate_uuid: "ba33a6c27967413fa2fb023978e615a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2fe31bca4040d2aec5437edfe48a59" is_pre_election: true
I20260620 08:18:12.075625 12418 raft_consensus.cc:2454] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba33a6c27967413fa2fb023978e615a2 in term 0.
I20260620 08:18:12.075614 12686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950beceab9204a488133467becb9f2af" candidate_uuid: "ba33a6c27967413fa2fb023978e615a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0100bbfa557482cac53fa6a752d09c0" is_pre_election: true
I20260620 08:18:12.075891 12686 raft_consensus.cc:2454] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba33a6c27967413fa2fb023978e615a2 in term 0.
I20260620 08:18:12.075990 12486 leader_election.cc:302] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2fe31bca4040d2aec5437edfe48a59, ba33a6c27967413fa2fb023978e615a2; no voters:
I20260620 08:18:12.076138 12754 raft_consensus.cc:2781] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:18:12.076195 12754 raft_consensus.cc:482] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:18:12.076231 12754 raft_consensus.cc:3037] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:12.076984 12754 raft_consensus.cc:504] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:12.077132 12754 leader_election.cc:288] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a2fe31bca4040d2aec5437edfe48a59 (127.24.249.129:34801), b0100bbfa557482cac53fa6a752d09c0 (127.24.249.131:39921)
I20260620 08:18:12.077298 12418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950beceab9204a488133467becb9f2af" candidate_uuid: "ba33a6c27967413fa2fb023978e615a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2fe31bca4040d2aec5437edfe48a59"
I20260620 08:18:12.077359 12686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950beceab9204a488133467becb9f2af" candidate_uuid: "ba33a6c27967413fa2fb023978e615a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0100bbfa557482cac53fa6a752d09c0"
I20260620 08:18:12.077402 12418 raft_consensus.cc:3037] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:12.077443 12686 raft_consensus.cc:3037] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:12.078186 12418 raft_consensus.cc:2454] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba33a6c27967413fa2fb023978e615a2 in term 1.
I20260620 08:18:12.078258 12686 raft_consensus.cc:2454] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba33a6c27967413fa2fb023978e615a2 in term 1.
I20260620 08:18:12.078397 12486 leader_election.cc:302] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2fe31bca4040d2aec5437edfe48a59, ba33a6c27967413fa2fb023978e615a2; no voters:
I20260620 08:18:12.078475 12754 raft_consensus.cc:2781] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:12.078626 12754 raft_consensus.cc:686] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 1 LEADER]: Becoming Leader. State: Replica: ba33a6c27967413fa2fb023978e615a2, State: Running, Role: LEADER
I20260620 08:18:12.078735 12754 consensus_queue.cc:237] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } }
I20260620 08:18:12.079581 12272 catalog_manager.cc:5495] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba33a6c27967413fa2fb023978e615a2 (127.24.249.130). New cstate: current_term: 1 leader_uuid: "ba33a6c27967413fa2fb023978e615a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba33a6c27967413fa2fb023978e615a2" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20260620 08:18:12.087338 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.084749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.24.249.130:44983)
0620 08:18:12.084798 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.084800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.084810 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:12.084884 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.084885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.084910 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.084994 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:12.085172 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.085661 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.086523 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.086720 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.086831 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.087018 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:12.087118 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:12.087215 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.087235 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.087255 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:18:12.089313 12686 raft_consensus.cc:1261] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 1 FOLLOWER]: Refusing update from remote peer ba33a6c27967413fa2fb023978e615a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:12.089313 12418 raft_consensus.cc:1261] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 1 FOLLOWER]: Refusing update from remote peer ba33a6c27967413fa2fb023978e615a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:12.089723 12759 consensus_queue.cc:1035] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0100bbfa557482cac53fa6a752d09c0" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:12.089922 12754 consensus_queue.cc:1035] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2fe31bca4040d2aec5437edfe48a59" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:12.092329 12762 mvcc.cc:205] Tried to move back new op lower bound from 7298840543594663936 to 7298840543554973696. Current Snapshot: MvccSnapshot[applied={T|T < 7298840543594663936}]
I20260620 08:18:12.092499 12764 mvcc.cc:205] Tried to move back new op lower bound from 7298840543594663936 to 7298840543554973696. Current Snapshot: MvccSnapshot[applied={T|T < 7298840543594663936}]
I20260620 08:18:12.093042 12763 mvcc.cc:205] Tried to move back new op lower bound from 7298840543594663936 to 7298840543554973696. Current Snapshot: MvccSnapshot[applied={T|T < 7298840543594663936 or (T in {7298840543594663936})}]
I20260620 08:18:12.094619 12273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:12.094708 12273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:12.095538 12273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c97bd37ae00c441fa9915d5478aaf241: Sending DeleteTablet for 3 replicas of tablet 950beceab9204a488133467becb9f2af
I20260620 08:18:12.095857 12532 tablet_service.cc:1514] Processing DeleteTablet for tablet 950beceab9204a488133467becb9f2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:12 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:34222
I20260620 08:18:12.095906 12398 tablet_service.cc:1514] Processing DeleteTablet for tablet 950beceab9204a488133467becb9f2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:12 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:49684
I20260620 08:18:12.095927 12666 tablet_service.cc:1514] Processing DeleteTablet for tablet 950beceab9204a488133467becb9f2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:12 UTC) from {username='kudu', principal='kudu/127.24.249.190@KRBTEST.COM'} at 127.0.0.1:49230
I20260620 08:18:12.096109 12773 tablet_replica.cc:330] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: stopping tablet replica
I20260620 08:18:12.096174 12772 tablet_replica.cc:330] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59: stopping tablet replica
I20260620 08:18:12.096276 12772 raft_consensus.cc:2229] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:12.096361 12773 raft_consensus.cc:2229] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:18:12.096402 12772 raft_consensus.cc:2258] T 950beceab9204a488133467becb9f2af P 8a2fe31bca4040d2aec5437edfe48a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:12.096366 12774 tablet_replica.cc:330] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: stopping tablet replica
I20260620 08:18:12.096467 12773 raft_consensus.cc:2258] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:12.096500 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 12335
I20260620 08:18:12.096879 12773 ts_tablet_manager.cc:1905] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:12.097213 12774 raft_consensus.cc:2229] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:12.097405 12774 raft_consensus.cc:2258] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:12.097977 12774 ts_tablet_manager.cc:1905] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:12.098533 12773 ts_tablet_manager.cc:1918] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:12.098594 12773 log.cc:1196] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/950beceab9204a488133467becb9f2af
I20260620 08:18:12.098795 12773 ts_tablet_manager.cc:1939] T 950beceab9204a488133467becb9f2af P ba33a6c27967413fa2fb023978e615a2: Deleting consensus metadata
I20260620 08:18:12.099143 12260 catalog_manager.cc:4841] TS ba33a6c27967413fa2fb023978e615a2 (127.24.249.130:44983): tablet 950beceab9204a488133467becb9f2af (table test-table [id=8af3abd4ecf64bacb625d9bad144b366]) successfully deleted
I20260620 08:18:12.100270 12774 ts_tablet_manager.cc:1918] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:12.100347 12774 log.cc:1196] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/950beceab9204a488133467becb9f2af
I20260620 08:18:12.100603 12774 ts_tablet_manager.cc:1939] T 950beceab9204a488133467becb9f2af P b0100bbfa557482cac53fa6a752d09c0: Deleting consensus metadata
I20260620 08:18:12.100992 12259 catalog_manager.cc:4841] TS b0100bbfa557482cac53fa6a752d09c0 (127.24.249.131:39921): tablet 950beceab9204a488133467becb9f2af (table test-table [id=8af3abd4ecf64bacb625d9bad144b366]) successfully deleted
W20260620 08:18:12.104182 12258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:34801 (error 108)
I20260620 08:18:12.104432 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 12469
I20260620 08:18:12.105046 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.104583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:34801 (local address 127.0.0.1:49688)
0620 08:18:12.104966 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.105007 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:34801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
W20260620 08:18:12.105183 12258 catalog_manager.cc:4584] TS 8a2fe31bca4040d2aec5437edfe48a59 (127.24.249.129:34801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 950beceab9204a488133467becb9f2af: Network error: Client connection negotiation failed: client connection to 127.24.249.129:34801: connect: Connection refused (error 111)
I20260620 08:18:12.109460 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 12603
I20260620 08:18:12.114636 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 12241
2026-06-20T08:18:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2145 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T08:18:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:12Z Disabled control of system clock
I20260620 08:18:12.143353 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42219
--webserver_interface=127.24.249.190
--webserver_port=0
--builtin_ntp_servers=127.24.249.148:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:12.226979 12779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:12.227267 12779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:12.227494 12779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:12.229557 12779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:12.229627 12779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:12.229642 12779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:12.229656 12779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:12.229673 12779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:12.231487 12779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:42219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:12.231736 12779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:12.232014 12779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:12.237787 12785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:12.237839 12779 server_base.cc:1034] running on GCE node
W20260620 08:18:12.237780 12784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:12.237905 12787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:12.238441 12779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:12.238821 12779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:12.240039 12779 hybrid_clock.cc:648] HybridClock initialized: now 1781943492240017 us; error 43 us; skew 500 ppm
I20260620 08:18:12.241488 12779 webserver.cc:460] Webserver started at http://127.24.249.190:35063/ using document root <none> and password file <none>
I20260620 08:18:12.241746 12779 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:12.241822 12779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:12.242007 12779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:12.243245 12779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "f808338d00b84e1e93764cf5a0eb9f97"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "e8f2f35c78b652d4de1664f50009fde2"
server_key_iv: "6cca9253c156c1cd3b71ecd85259a9df"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.243661 12779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "f808338d00b84e1e93764cf5a0eb9f97"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "e8f2f35c78b652d4de1664f50009fde2"
server_key_iv: "6cca9253c156c1cd3b71ecd85259a9df"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.245085 12779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:12.246016 12793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.246258 12779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:12.246345 12779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "f808338d00b84e1e93764cf5a0eb9f97"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "e8f2f35c78b652d4de1664f50009fde2"
server_key_iv: "6cca9253c156c1cd3b71ecd85259a9df"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.246405 12779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:12.260583 12779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:12.260876 12779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:12.260982 12779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:12.265444 12779 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:42219
I20260620 08:18:12.265517 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:42219 every 8 connection(s)
I20260620 08:18:12.265841 12779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:12.267017 12846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:12.268435 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 12779
I20260620 08:18:12.268528 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:12.268764 25574 external_mini_cluster.cc:1428] Setting key c2d8d976529c78fef43c4edf2a23d7c8
I20260620 08:18:12.269819 12846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97: Bootstrap starting.
I20260620 08:18:12.270576 12846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:12.270885 12846 log.cc:824] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:12.271575 12846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97: No bootstrap required, opened a new log
I20260620 08:18:12.273360 12846 raft_consensus.cc:348] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808338d00b84e1e93764cf5a0eb9f97" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42219 } }
I20260620 08:18:12.273528 12846 raft_consensus.cc:374] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:12.273579 12846 raft_consensus.cc:729] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f808338d00b84e1e93764cf5a0eb9f97, State: Initialized, Role: FOLLOWER
I20260620 08:18:12.273722 12846 consensus_queue.cc:260] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808338d00b84e1e93764cf5a0eb9f97" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42219 } }
I20260620 08:18:12.273820 12846 raft_consensus.cc:388] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:12.273882 12846 raft_consensus.cc:482] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:12.273931 12846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:12.274471 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.269375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46468 (local address 127.24.249.190:42219)
0620 08:18:12.269566 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.269573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.269597 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:18:12.270106 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.270114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.270177 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.270298 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:12.270582 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.271592 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.272421 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.272847 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.272993 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.273692 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:12.273697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:12.273697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:12.273942 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:12.273988 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:12.274005 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:12.274075 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.274165 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.274253 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260620 08:18:12.274685 12846 raft_consensus.cc:504] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808338d00b84e1e93764cf5a0eb9f97" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42219 } }
I20260620 08:18:12.274821 12846 leader_election.cc:302] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f808338d00b84e1e93764cf5a0eb9f97; no voters:
I20260620 08:18:12.275014 12846 leader_election.cc:288] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:12.275063 12851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:12.275295 12846 sys_catalog.cc:564] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:12.275403 12851 raft_consensus.cc:686] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [term 1 LEADER]: Becoming Leader. State: Replica: f808338d00b84e1e93764cf5a0eb9f97, State: Running, Role: LEADER
I20260620 08:18:12.275499 12851 consensus_queue.cc:237] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808338d00b84e1e93764cf5a0eb9f97" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42219 } }
I20260620 08:18:12.275892 12853 sys_catalog.cc:455] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f808338d00b84e1e93764cf5a0eb9f97. Latest consensus state: current_term: 1 leader_uuid: "f808338d00b84e1e93764cf5a0eb9f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808338d00b84e1e93764cf5a0eb9f97" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42219 } } }
I20260620 08:18:12.275985 12853 sys_catalog.cc:458] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:12.275905 12852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f808338d00b84e1e93764cf5a0eb9f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808338d00b84e1e93764cf5a0eb9f97" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 42219 } } }
I20260620 08:18:12.276026 12852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:12.276297 12860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:12.276870 12860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:12.278409 12860 catalog_manager.cc:1347] Generated new cluster ID: 99eb091a930d427bb3b15787786161fd
I20260620 08:18:12.278462 12860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:12.281697 12860 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:12.282232 12860 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:12.287681 12860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97: Generated new TSK 0
I20260620 08:18:12.287874 12860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:12.293972 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.129:0
--local_ip_for_outbound_sockets=127.24.249.129
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42219
--builtin_ntp_servers=127.24.249.148:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:12.382501 12870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:12.382743 12870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:12.382777 12870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:12.383977 12870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:12.384040 12870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:12.384084 12870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.129
I20260620 08:18:12.385502 12870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.249.129
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.129
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:12.385769 12870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:12.386124 12870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:12.392656 12875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:12.392722 12878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:12.392843 12870 server_base.cc:1034] running on GCE node
W20260620 08:18:12.392992 12876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:12.393234 12870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:12.393545 12870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:12.394747 12870 hybrid_clock.cc:648] HybridClock initialized: now 1781943492394727 us; error 33 us; skew 500 ppm
I20260620 08:18:12.396207 12870 webserver.cc:460] Webserver started at http://127.24.249.129:33757/ using document root <none> and password file <none>
I20260620 08:18:12.396485 12870 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:12.396561 12870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:12.396744 12870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:12.397982 12870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data/instance:
uuid: "9ff0415b254642d7bc63b94284674448"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "1635beab37090eb47c118121ec17c65d"
server_key_iv: "199886c1cc64f703fd324acec4aa1aeb"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.398365 12870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance:
uuid: "9ff0415b254642d7bc63b94284674448"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "1635beab37090eb47c118121ec17c65d"
server_key_iv: "199886c1cc64f703fd324acec4aa1aeb"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.399848 12870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:12.401077 12884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.401288 12870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:12.401378 12870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/wal
uuid: "9ff0415b254642d7bc63b94284674448"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "1635beab37090eb47c118121ec17c65d"
server_key_iv: "199886c1cc64f703fd324acec4aa1aeb"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.401463 12870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:12.419132 12870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:12.419497 12870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:12.419657 12870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:12.419919 12870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:12.420276 12870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:12.420346 12870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.420392 12870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:12.420423 12870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.426744 12870 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.129:41113
I20260620 08:18:12.426800 12997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.129:41113 every 8 connection(s)
I20260620 08:18:12.427152 12870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/data/info.pb
I20260620 08:18:12.431414 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 12870
I20260620 08:18:12.431495 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-0/wal/instance
I20260620 08:18:12.431723 25574 external_mini_cluster.cc:1428] Setting key 3c1f94811d23249e563bab0bc63dec77
I20260620 08:18:12.432818 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.130:0
--local_ip_for_outbound_sockets=127.24.249.130
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42219
--builtin_ntp_servers=127.24.249.148:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 08:18:12.433065 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.428039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.129:60019 (local address 127.24.249.190:42219)
0620 08:18:12.428129 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.428131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.428330 (+ 199us) server_negotiation.cc:407] Connection header received
0620 08:18:12.428828 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.428829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.428846 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.428915 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:12.429606 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.430207 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.431411 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.431672 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.431762 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.432725 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:12.432727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:12.432727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:12.432732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:12.432750 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:12.432760 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:12.432838 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.432964 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.432996 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 08:18:12.433495 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.428003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42219 (local address 127.24.249.129:60019)
0620 08:18:12.428209 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.428243 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.428728 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.428966 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.428969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.428978 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:12.429500 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.429505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.430462 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.430463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.431277 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.431279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.431775 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.431776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.431955 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.432582 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:12.432583 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:12.432613 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:12.432638 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:12.432842 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:12.432849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:12.432852 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.433179 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.433290 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260620 08:18:12.434075 12998 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42219
I20260620 08:18:12.434167 12998 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:12.434365 12998 heartbeater.cc:511] Master 127.24.249.190:42219 requested a full tablet report, sending...
I20260620 08:18:12.434966 12810 ts_manager.cc:194] Registered new tserver with Master: 9ff0415b254642d7bc63b94284674448 (127.24.249.129:41113)
I20260620 08:18:12.436025 12810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.129:60019
W20260620 08:18:12.519654 13001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:12.519878 13001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:12.519902 13001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:12.521131 13001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:12.521175 13001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:12.521201 13001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.130
I20260620 08:18:12.522677 13001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.249.130
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.130
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:12.523242 13001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:12.523592 13001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:12.529238 13009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:12.529353 13006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:12.529345 13007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:12.529539 13001 server_base.cc:1034] running on GCE node
I20260620 08:18:12.529827 13001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:12.530124 13001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:12.531299 13001 hybrid_clock.cc:648] HybridClock initialized: now 1781943492531284 us; error 29 us; skew 500 ppm
I20260620 08:18:12.533295 13001 webserver.cc:460] Webserver started at http://127.24.249.130:44933/ using document root <none> and password file <none>
I20260620 08:18:12.533686 13001 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:12.533751 13001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:12.533962 13001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:12.535338 13001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data/instance:
uuid: "c62c362c41c14a00adb5288d6f85d7e3"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "4efd291d0baf535b07bc754d87f9e43a"
server_key_iv: "ee06bf3aec9d1e637056a07aaab7d21f"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.536190 13001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance:
uuid: "c62c362c41c14a00adb5288d6f85d7e3"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "4efd291d0baf535b07bc754d87f9e43a"
server_key_iv: "ee06bf3aec9d1e637056a07aaab7d21f"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.539053 13001 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260620 08:18:12.540392 13015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.540741 13001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:12.540894 13001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal
uuid: "c62c362c41c14a00adb5288d6f85d7e3"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "4efd291d0baf535b07bc754d87f9e43a"
server_key_iv: "ee06bf3aec9d1e637056a07aaab7d21f"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.541100 13001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:12.563117 13001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:12.563529 13001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:12.563690 13001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:12.563941 13001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:12.564311 13001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:12.564365 13001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.564409 13001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:12.564437 13001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.571301 13001 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.130:46805
I20260620 08:18:12.571432 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.130:46805 every 8 connection(s)
I20260620 08:18:12.571735 13001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/data/info.pb
I20260620 08:18:12.579238 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.572679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:52881 (local address 127.24.249.190:42219)
0620 08:18:12.572737 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.572739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.572935 (+ 196us) server_negotiation.cc:407] Connection header received
0620 08:18:12.573477 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.573479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.573498 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.573628 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:12.574693 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.575379 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.577185 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.577389 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.577500 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.578654 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:12.578655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:12.578656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:12.578661 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:12.578678 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:12.578682 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:12.578744 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.579052 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.579079 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 08:18:12.579262 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.572492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42219 (local address 127.24.249.130:52881)
0620 08:18:12.572686 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.572730 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.573290 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.573818 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.573825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.573837 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:12.574525 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.574534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.575653 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.575655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.577031 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.577036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.577546 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.577547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.577667 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.578391 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:12.578392 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:12.578424 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:12.578456 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:12.578779 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:12.578786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:12.578789 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.578948 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.579056 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260620 08:18:12.580310 13129 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42219
I20260620 08:18:12.580338 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13001
I20260620 08:18:12.580400 13129 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:12.580475 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal/instance
I20260620 08:18:12.580634 13129 heartbeater.cc:511] Master 127.24.249.190:42219 requested a full tablet report, sending...
I20260620 08:18:12.580700 25574 external_mini_cluster.cc:1428] Setting key 64d70337218579712d965f67add3ce10
I20260620 08:18:12.581604 12810 ts_manager.cc:194] Registered new tserver with Master: c62c362c41c14a00adb5288d6f85d7e3 (127.24.249.130:46805)
I20260620 08:18:12.582186 12810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.130:52881
I20260620 08:18:12.582597 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.249.131:0
--local_ip_for_outbound_sockets=127.24.249.131
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42219
--builtin_ntp_servers=127.24.249.148:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 08:18:12.672520 13132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:12.672782 13132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:12.672817 13132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:12.674049 13132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:12.674109 13132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:12.674155 13132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.249.131
I20260620 08:18:12.675714 13132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.249.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.249.131
--webserver_port=0
--tserver_master_addrs=127.24.249.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.249.131
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:12.675954 13132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:12.676213 13132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:12.682906 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:12.683012 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:12.683199 13132 server_base.cc:1034] running on GCE node
W20260620 08:18:12.682996 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:12.683748 13132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:12.684034 13132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:12.685220 13132 hybrid_clock.cc:648] HybridClock initialized: now 1781943492685205 us; error 52 us; skew 500 ppm
I20260620 08:18:12.687042 13132 webserver.cc:460] Webserver started at http://127.24.249.131:46395/ using document root <none> and password file <none>
I20260620 08:18:12.687333 13132 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:12.687409 13132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:12.687566 13132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:12.688989 13132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data/instance:
uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "2587e5d3b362b734cf018211591c1a40"
server_key_iv: "060c4d0fb1833f62091a84813d775d1c"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.689373 13132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance:
uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "2587e5d3b362b734cf018211591c1a40"
server_key_iv: "060c4d0fb1833f62091a84813d775d1c"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.690989 13132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:18:12.692202 13146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.692595 13132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:12.692724 13132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal
uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272"
format_stamp: "Formatted at 2026-06-20 08:18:12 on dist-test-slave-65w8"
server_key: "2587e5d3b362b734cf018211591c1a40"
server_key_iv: "060c4d0fb1833f62091a84813d775d1c"
server_key_version: "encryptionkey@0"
I20260620 08:18:12.692916 13132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:12.704851 13132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:12.705210 13132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:12.705399 13132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:12.705646 13132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 08:18:12.706030 13132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 08:18:12.706089 13132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.706132 13132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 08:18:12.706159 13132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:12.712302 13132 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.131:45483
I20260620 08:18:12.712352 13259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.131:45483 every 8 connection(s)
I20260620 08:18:12.712687 13132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/data/info.pb
I20260620 08:18:12.718222 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.713388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.131:34155 (local address 127.24.249.190:42219)
0620 08:18:12.713464 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.713466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.713758 (+ 292us) server_negotiation.cc:407] Connection header received
0620 08:18:12.714262 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.714263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.714273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.714332 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:12.715099 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.715756 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.716763 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.716956 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.717014 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.717963 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:12.717964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:12.717964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:12.717970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:12.717985 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:12.717989 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:12.718045 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.718140 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.718162 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:18:12.718509 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13132
I20260620 08:18:12.718624 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal/instance
I20260620 08:18:12.718832 25574 external_mini_cluster.cc:1428] Setting key 0fadcff999489d1ee52ba83b7336306a
I20260620 08:18:12.718827 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.713378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:42219 (local address 127.24.249.131:34155)
0620 08:18:12.713592 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.713654 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.714172 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.714382 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.714385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.714393 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:12.714883 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.714887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.715865 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.715866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.716666 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.716668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.717040 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.717041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.717149 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.717783 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 08:18:12.717784 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:12.717812 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 08:18:12.717836 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:12.718063 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:12.718068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:12.718071 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.718129 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.718214 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
I20260620 08:18:12.719427 13260 heartbeater.cc:347] Connected to a master server at 127.24.249.190:42219
I20260620 08:18:12.719545 13260 heartbeater.cc:464] Registering TS with master...
I20260620 08:18:12.719725 13260 heartbeater.cc:511] Master 127.24.249.190:42219 requested a full tablet report, sending...
I20260620 08:18:12.720140 12810 ts_manager.cc:194] Registered new tserver with Master: ef2c00fb2fde4b9ba3520cf67c2ac272 (127.24.249.131:45483)
I20260620 08:18:12.720517 12810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.249.131:34155
I20260620 08:18:12.731173 25574 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 08:18:12.735929 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.732642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.24.249.190:42219)
0620 08:18:12.732739 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.732741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.732862 (+ 121us) server_negotiation.cc:407] Connection header received
0620 08:18:12.732998 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.732998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.733008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.733075 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:12.733254 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.733786 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.734653 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.734847 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.734921 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.735661 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:12.735662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:12.735663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:12.735667 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:12.735681 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:12.735685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:12.735754 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.735808 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.735834 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 08:18:12.737972 12810 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 08:18:12.740018 12810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 08:18:12.747239 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.742837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.24.249.131:45483)
0620 08:18:12.743092 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.743096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.743103 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:18:12.743137 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.743138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.743150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.743259 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:12.743407 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.744563 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.746112 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.746814 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.746894 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.746985 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.747098 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.747149 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260620 08:18:12.747267 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.742739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:45483 (local address 127.0.0.1:59560)
0620 08:18:12.742904 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.742913 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.742951 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.743269 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.743270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.743272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:12.743322 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.743324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.744640 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.744641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.745997 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.746000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.746905 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.746906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.746984 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.746998 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.747082 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.747140 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":61,"thread_start_us":112,"threads_started":1}
I20260620 08:18:12.747287 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.742537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.130:46805 (local address 127.0.0.1:54030)
0620 08:18:12.742833 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.742875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.742955 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.743614 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.743615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.743617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:12.743652 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.743653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.744524 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.744525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.745974 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.745977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.747023 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.747024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.747095 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.747100 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.747149 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.747190 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":29,"thread_start_us":59,"threads_started":1}
I20260620 08:18:12.747500 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.742630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.24.249.130:46805)
0620 08:18:12.743461 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.743465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.743487 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:18:12.743527 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.743528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.743543 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.743617 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:12.743706 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.744437 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.746061 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.746815 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.746892 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.746985 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.747327 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.747385 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":89,"threads_started":1}
I20260620 08:18:12.748042 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.742553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:41113 (local address 127.0.0.1:39720)
0620 08:18:12.742845 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.742875 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.742948 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.743215 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.743236 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.743239 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:12.743319 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.743321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.745681 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.745682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.746486 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.746487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.747700 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.747701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.747803 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.747807 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.747851 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.747888 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":99,"threads_started":1}
I20260620 08:18:12.748216 13194 tablet_service.cc:1467] Processing CreateTablet for tablet febb5384a74d48c8a84d21b4f483268b (DEFAULT_TABLE table=test-table [id=455dc88f511d498aa4f86a087a30e581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:12.748346 13063 tablet_service.cc:1467] Processing CreateTablet for tablet febb5384a74d48c8a84d21b4f483268b (DEFAULT_TABLE table=test-table [id=455dc88f511d498aa4f86a087a30e581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:12.748585 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.742656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.24.249.129:41113)
0620 08:18:12.743051 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.743055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.743065 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:12.743110 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.743112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.743124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.743187 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:12.743408 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.744556 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.746573 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.747229 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.747297 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.748365 (+ 1068us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.748392 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.748436 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":76,"threads_started":1}
I20260620 08:18:12.748610 13194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet febb5384a74d48c8a84d21b4f483268b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:12.748705 13063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet febb5384a74d48c8a84d21b4f483268b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:12.749223 12932 tablet_service.cc:1467] Processing CreateTablet for tablet febb5384a74d48c8a84d21b4f483268b (DEFAULT_TABLE table=test-table [id=455dc88f511d498aa4f86a087a30e581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 08:18:12.749686 12932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet febb5384a74d48c8a84d21b4f483268b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:12.751845 13274 tablet_bootstrap.cc:492] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Bootstrap starting.
I20260620 08:18:12.752363 13275 tablet_bootstrap.cc:492] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Bootstrap starting.
I20260620 08:18:12.752557 13274 tablet_bootstrap.cc:654] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:12.752974 13274 log.cc:824] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:12.753043 13275 tablet_bootstrap.cc:654] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:12.753254 13276 tablet_bootstrap.cc:492] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448: Bootstrap starting.
I20260620 08:18:12.753355 13275 log.cc:824] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:12.753926 13274 tablet_bootstrap.cc:492] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: No bootstrap required, opened a new log
I20260620 08:18:12.753968 13276 tablet_bootstrap.cc:654] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:12.754005 13274 ts_tablet_manager.cc:1397] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 08:18:12.754168 13275 tablet_bootstrap.cc:492] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: No bootstrap required, opened a new log
I20260620 08:18:12.754228 13275 ts_tablet_manager.cc:1397] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 08:18:12.754267 13276 log.cc:824] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:12.754995 13276 tablet_bootstrap.cc:492] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448: No bootstrap required, opened a new log
I20260620 08:18:12.755067 13276 ts_tablet_manager.cc:1397] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 08:18:12.756129 13275 raft_consensus.cc:348] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.756294 13275 raft_consensus.cc:374] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:12.756340 13275 raft_consensus.cc:729] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62c362c41c14a00adb5288d6f85d7e3, State: Initialized, Role: FOLLOWER
I20260620 08:18:12.756479 13275 consensus_queue.cc:260] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.756736 13275 ts_tablet_manager.cc:1428] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 08:18:12.756716 13274 raft_consensus.cc:348] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.756851 13129 heartbeater.cc:503] Master 127.24.249.190:42219 was elected leader, sending a full tablet report...
I20260620 08:18:12.756866 13274 raft_consensus.cc:374] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:12.756912 13274 raft_consensus.cc:729] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2c00fb2fde4b9ba3520cf67c2ac272, State: Initialized, Role: FOLLOWER
I20260620 08:18:12.757026 13274 consensus_queue.cc:260] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.757268 13274 ts_tablet_manager.cc:1428] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 08:18:12.757400 13260 heartbeater.cc:503] Master 127.24.249.190:42219 was elected leader, sending a full tablet report...
I20260620 08:18:12.757570 13276 raft_consensus.cc:348] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.757731 13276 raft_consensus.cc:374] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:12.757779 13276 raft_consensus.cc:729] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff0415b254642d7bc63b94284674448, State: Initialized, Role: FOLLOWER
I20260620 08:18:12.757916 13276 consensus_queue.cc:260] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.758188 13276 ts_tablet_manager.cc:1428] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 08:18:12.758297 12998 heartbeater.cc:503] Master 127.24.249.190:42219 was elected leader, sending a full tablet report...
W20260620 08:18:12.822371 13130 tablet.cc:2369] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:18:12.921175 13280 raft_consensus.cc:482] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 08:18:12.921309 13280 raft_consensus.cc:504] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.921665 13280 leader_election.cc:288] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ff0415b254642d7bc63b94284674448 (127.24.249.129:41113), ef2c00fb2fde4b9ba3520cf67c2ac272 (127.24.249.131:45483)
I20260620 08:18:12.925684 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.921980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:39461 (local address 127.24.249.129:41113)
0620 08:18:12.922031 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.922033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.922048 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:18:12.922082 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.922083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.922097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.922189 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:12.922465 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.923232 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.924725 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.925267 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.925352 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.925359 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.925564 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.925607 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:18:12.925684 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.922007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.249.130:38889 (local address 127.24.249.131:45483)
0620 08:18:12.922052 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.922054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.922154 (+ 100us) server_negotiation.cc:407] Connection header received
0620 08:18:12.922211 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.922211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.922222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.922278 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 08:18:12.922464 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.923251 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.924724 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.925284 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.925352 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.925360 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.925564 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.925608 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 08:18:12.925724 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.921862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:41113 (local address 127.24.249.130:39461)
0620 08:18:12.921940 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.921954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.922008 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.922204 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.922205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.922208 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:12.922301 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.922303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.923349 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.923351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.924630 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.924633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.925365 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.925366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.925481 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.925486 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.925546 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.925612 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":19}
I20260620 08:18:12.925729 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.921906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.131:45483 (local address 127.24.249.130:38889)
0620 08:18:12.922075 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.922083 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:12.922137 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:12.922299 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:12.922300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:12.922301 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 08:18:12.922361 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.922362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.923350 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.923351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.924630 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:12.924633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.925365 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.925366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:12.925481 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.925486 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 08:18:12.925546 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:18:12.925610 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":15,"thread_start_us":101,"threads_started":1}
I20260620 08:18:12.926100 12952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febb5384a74d48c8a84d21b4f483268b" candidate_uuid: "c62c362c41c14a00adb5288d6f85d7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff0415b254642d7bc63b94284674448" is_pre_election: true
I20260620 08:18:12.926270 12952 raft_consensus.cc:2454] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c62c362c41c14a00adb5288d6f85d7e3 in term 0.
I20260620 08:18:12.926513 13214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febb5384a74d48c8a84d21b4f483268b" candidate_uuid: "c62c362c41c14a00adb5288d6f85d7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" is_pre_election: true
I20260620 08:18:12.926559 13019 leader_election.cc:302] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff0415b254642d7bc63b94284674448, c62c362c41c14a00adb5288d6f85d7e3; no voters:
I20260620 08:18:12.926657 13214 raft_consensus.cc:2454] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c62c362c41c14a00adb5288d6f85d7e3 in term 0.
I20260620 08:18:12.926681 13280 raft_consensus.cc:2781] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 08:18:12.926748 13280 raft_consensus.cc:482] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 08:18:12.926786 13280 raft_consensus.cc:3037] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:12.927520 13280 raft_consensus.cc:504] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.927675 13280 leader_election.cc:288] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ff0415b254642d7bc63b94284674448 (127.24.249.129:41113), ef2c00fb2fde4b9ba3520cf67c2ac272 (127.24.249.131:45483)
I20260620 08:18:12.927837 13214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febb5384a74d48c8a84d21b4f483268b" candidate_uuid: "c62c362c41c14a00adb5288d6f85d7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272"
I20260620 08:18:12.927832 12952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "febb5384a74d48c8a84d21b4f483268b" candidate_uuid: "c62c362c41c14a00adb5288d6f85d7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff0415b254642d7bc63b94284674448"
I20260620 08:18:12.927944 12952 raft_consensus.cc:3037] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [term 0 FOLLOWER]: Advancing to term 1
W20260620 08:18:12.927989 12999 tablet.cc:2369] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:18:12.927928 13214 raft_consensus.cc:3037] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:12.928762 12952 raft_consensus.cc:2454] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c62c362c41c14a00adb5288d6f85d7e3 in term 1.
I20260620 08:18:12.928848 13214 raft_consensus.cc:2454] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c62c362c41c14a00adb5288d6f85d7e3 in term 1.
I20260620 08:18:12.928970 13019 leader_election.cc:302] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ff0415b254642d7bc63b94284674448, c62c362c41c14a00adb5288d6f85d7e3; no voters:
I20260620 08:18:12.929090 13280 raft_consensus.cc:2781] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:12.929282 13280 raft_consensus.cc:686] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 1 LEADER]: Becoming Leader. State: Replica: c62c362c41c14a00adb5288d6f85d7e3, State: Running, Role: LEADER
I20260620 08:18:12.929419 13280 consensus_queue.cc:237] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } }
I20260620 08:18:12.930352 12810 catalog_manager.cc:5495] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c62c362c41c14a00adb5288d6f85d7e3 (127.24.249.130). New cstate: current_term: 1 leader_uuid: "c62c362c41c14a00adb5288d6f85d7e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c62c362c41c14a00adb5288d6f85d7e3" member_type: VOTER last_known_addr { host: "127.24.249.130" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 } health_report { overall_health: UNKNOWN } } }
W20260620 08:18:12.963251 13261 tablet.cc:2369] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 08:18:12.973986 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:12.971535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.24.249.130:46805)
0620 08:18:12.971612 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:12.971614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:12.971629 (+ 15us) server_negotiation.cc:407] Connection header received
0620 08:18:12.971666 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:12.971667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:12.971689 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:12.971768 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:12.971909 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.972406 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.973167 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:12.973364 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:12.973441 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:12.973613 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:12.973712 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:12.973805 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:12.973822 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 08:18:12.973843 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 08:18:12.976019 13214 raft_consensus.cc:1261] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 1 FOLLOWER]: Refusing update from remote peer c62c362c41c14a00adb5288d6f85d7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:12.976029 12952 raft_consensus.cc:1261] T febb5384a74d48c8a84d21b4f483268b P 9ff0415b254642d7bc63b94284674448 [term 1 FOLLOWER]: Refusing update from remote peer c62c362c41c14a00adb5288d6f85d7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:12.976361 13280 consensus_queue.cc:1035] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2c00fb2fde4b9ba3520cf67c2ac272" member_type: VOTER last_known_addr { host: "127.24.249.131" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:12.976608 13284 consensus_queue.cc:1035] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff0415b254642d7bc63b94284674448" member_type: VOTER last_known_addr { host: "127.24.249.129" port: 41113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:12.978857 13287 mvcc.cc:205] Tried to move back new op lower bound from 7298840547226374144 to 7298840547039535104. Current Snapshot: MvccSnapshot[applied={T|T < 7298840547226374144}]
I20260620 08:18:12.979146 13289 mvcc.cc:205] Tried to move back new op lower bound from 7298840547226374144 to 7298840547039535104. Current Snapshot: MvccSnapshot[applied={T|T < 7298840547226374144}]
I20260620 08:18:12.980021 13288 mvcc.cc:205] Tried to move back new op lower bound from 7298840547226374144 to 7298840547039535104. Current Snapshot: MvccSnapshot[applied={T|T < 7298840547226374144}]
I20260620 08:18:12.981465 12810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:12.981544 12810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 08:18:12.982313 12810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f808338d00b84e1e93764cf5a0eb9f97: Sending DeleteTablet for 3 replicas of tablet febb5384a74d48c8a84d21b4f483268b
I20260620 08:18:12.982700 13063 tablet_service.cc:1514] Processing DeleteTablet for tablet febb5384a74d48c8a84d21b4f483268b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:12 UTC) from {username='slave'} at 127.0.0.1:54030
I20260620 08:18:12.982698 13194 tablet_service.cc:1514] Processing DeleteTablet for tablet febb5384a74d48c8a84d21b4f483268b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:12 UTC) from {username='slave'} at 127.0.0.1:59560
I20260620 08:18:12.982896 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 12870
I20260620 08:18:12.982915 13297 tablet_replica.cc:330] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: stopping tablet replica
I20260620 08:18:12.982929 12932 tablet_service.cc:1514] Processing DeleteTablet for tablet febb5384a74d48c8a84d21b4f483268b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 08:18:12 UTC) from {username='slave'} at 127.0.0.1:39720
I20260620 08:18:12.983012 13297 raft_consensus.cc:2229] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 08:18:12.983124 13297 raft_consensus.cc:2258] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:12.984259 13297 ts_tablet_manager.cc:1905] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:12.985966 13297 ts_tablet_manager.cc:1918] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:12.986040 13297 log.cc:1196] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-1/wal/wals/febb5384a74d48c8a84d21b4f483268b
I20260620 08:18:12.986301 13297 ts_tablet_manager.cc:1939] T febb5384a74d48c8a84d21b4f483268b P c62c362c41c14a00adb5288d6f85d7e3: Deleting consensus metadata
I20260620 08:18:12.986733 12795 catalog_manager.cc:4841] TS c62c362c41c14a00adb5288d6f85d7e3 (127.24.249.130:46805): tablet febb5384a74d48c8a84d21b4f483268b (table test-table [id=455dc88f511d498aa4f86a087a30e581]) successfully deleted
W20260620 08:18:12.988919 12797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.249.129:41113 (error 108)
W20260620 08:18:12.989451 13268 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:12.989294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.129:41113 (local address 127.0.0.1:39722)
0620 08:18:12.989359 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:12.989404 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.129:41113: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:18:12.989569 13298 tablet_replica.cc:330] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: stopping tablet replica
I20260620 08:18:12.989574 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13001
I20260620 08:18:12.989694 13298 raft_consensus.cc:2229] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 08:18:12.989797 13298 raft_consensus.cc:2258] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 08:18:12.990592 13298 ts_tablet_manager.cc:1905] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 08:18:12.992149 13298 ts_tablet_manager.cc:1918] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 08:18:12.992219 13298 log.cc:1196] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Deleting WAL directory at /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781943243671421-25574-0/minicluster-data/ts-2/wal/wals/febb5384a74d48c8a84d21b4f483268b
W20260620 08:18:12.992470 12797 catalog_manager.cc:4584] TS 9ff0415b254642d7bc63b94284674448 (127.24.249.129:41113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet febb5384a74d48c8a84d21b4f483268b: Network error: Client connection negotiation failed: client connection to 127.24.249.129:41113: connect: Connection reset by peer (error 104)
I20260620 08:18:12.992532 13298 ts_tablet_manager.cc:1939] T febb5384a74d48c8a84d21b4f483268b P ef2c00fb2fde4b9ba3520cf67c2ac272: Deleting consensus metadata
I20260620 08:18:12.993022 12795 catalog_manager.cc:4841] TS ef2c00fb2fde4b9ba3520cf67c2ac272 (127.24.249.131:45483): tablet febb5384a74d48c8a84d21b4f483268b (table test-table [id=455dc88f511d498aa4f86a087a30e581]) successfully deleted
I20260620 08:18:12.996025 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13132
I20260620 08:18:13.001694 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 12779
2026-06-20T08:18:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (886 ms)
[----------] 6 tests from EncryptionPolicyTest (8559 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:13 dist-test-slave-65w8 krb5kdc[13302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:13 dist-test-slave-65w8 krb5kdc[13302](info): set up 2 sockets
Jun 20 08:18:13 dist-test-slave-65w8 krb5kdc[13302](info): commencing operation
krb5kdc: starting...
W20260620 08:18:14.058012 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:14 dist-test-slave-65w8 krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:14.185843 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:14.268918 13315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:14.269152 13315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:14.269177 13315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:14.270368 13315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:14.270413 13315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:14.270427 13315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:14.270439 13315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:14.270452 13315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:14.272197 13315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:14.272433 13315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:14.272616 13315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:14.277628 13323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:14.277769 13321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:14.277622 13320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:14.277937 13315 server_base.cc:1034] running on GCE node
I20260620 08:18:14.278216 13315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:14.278503 13315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:14.279663 13315 hybrid_clock.cc:648] HybridClock initialized: now 1781943494279661 us; error 38 us; skew 500 ppm
Jun 20 08:18:14 dist-test-slave-65w8 krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:14.282323 13315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:14.282799 13315 webserver.cc:460] Webserver started at http://127.0.0.1:45053/ using document root <none> and password file <none>
I20260620 08:18:14.283022 13315 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:14.283078 13315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:14.283254 13315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:14.284368 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "32eb1c48770549f5964f60cff8533200"
format_stamp: "Formatted at 2026-06-20 08:18:14 on dist-test-slave-65w8"
server_key: "e29720e3a80ce86ce5d0e950c0c4bf61"
server_key_iv: "b03c691aeab03bbaaad70f2d4201bcd2"
server_key_version: "encryptionkey@0"
I20260620 08:18:14.284734 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "32eb1c48770549f5964f60cff8533200"
format_stamp: "Formatted at 2026-06-20 08:18:14 on dist-test-slave-65w8"
server_key: "e29720e3a80ce86ce5d0e950c0c4bf61"
server_key_iv: "b03c691aeab03bbaaad70f2d4201bcd2"
server_key_version: "encryptionkey@0"
I20260620 08:18:14.286163 13315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:18:14.286984 13330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:14.287181 13315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:14.287263 13315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "32eb1c48770549f5964f60cff8533200"
format_stamp: "Formatted at 2026-06-20 08:18:14 on dist-test-slave-65w8"
server_key: "e29720e3a80ce86ce5d0e950c0c4bf61"
server_key_iv: "b03c691aeab03bbaaad70f2d4201bcd2"
server_key_version: "encryptionkey@0"
I20260620 08:18:14.287310 13315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:14.305249 13315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:14.307451 13315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:14.307560 13315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:14.311625 13315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42893
I20260620 08:18:14.311678 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42893 every 8 connection(s)
I20260620 08:18:14.312175 13315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:14.312911 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:14.315218 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200: Bootstrap starting.
I20260620 08:18:14.315796 13383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:14.316089 13383 log.cc:824] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:14.316812 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200: No bootstrap required, opened a new log
I20260620 08:18:14.318549 13383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eb1c48770549f5964f60cff8533200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260620 08:18:14.318708 13383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:14.318760 13383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32eb1c48770549f5964f60cff8533200, State: Initialized, Role: FOLLOWER
I20260620 08:18:14.318946 13383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eb1c48770549f5964f60cff8533200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260620 08:18:14.319047 13383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:14.319092 13383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:14.319144 13383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:14.320066 13383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eb1c48770549f5964f60cff8533200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260620 08:18:14.320186 13383 leader_election.cc:302] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32eb1c48770549f5964f60cff8533200; no voters:
I20260620 08:18:14.320382 13383 leader_election.cc:288] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:14.320422 13386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:14.320603 13386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [term 1 LEADER]: Becoming Leader. State: Replica: 32eb1c48770549f5964f60cff8533200, State: Running, Role: LEADER
I20260620 08:18:14.320679 13383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:14.320734 13386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eb1c48770549f5964f60cff8533200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } }
I20260620 08:18:14.321130 13388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32eb1c48770549f5964f60cff8533200. Latest consensus state: current_term: 1 leader_uuid: "32eb1c48770549f5964f60cff8533200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eb1c48770549f5964f60cff8533200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } } }
I20260620 08:18:14.321223 13388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:14.321117 13387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32eb1c48770549f5964f60cff8533200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eb1c48770549f5964f60cff8533200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42893 } } }
I20260620 08:18:14.321388 13387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:14.321727 13394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:14.321803 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13315
I20260620 08:18:14.321952 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:14.322185 25574 external_mini_cluster.cc:1428] Setting key c8bd0ac98226c246cffac37aeaee954b
I20260620 08:18:14.322294 13394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:14.323832 13394 catalog_manager.cc:1347] Generated new cluster ID: 287e4fcd23714460869fb9a4e9e41b03
I20260620 08:18:14.323885 13394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:18:14 dist-test-slave-65w8 krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:18:14.332530 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:14.322676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33427 (local address 127.0.0.1:42893)
0620 08:18:14.322852 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:14.322861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:14.322915 (+ 54us) server_negotiation.cc:407] Connection header received
0620 08:18:14.323032 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:14.323039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:14.323093 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:14.323271 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:14.323926 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:14.324834 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:14.325596 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:14.326120 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:14.326233 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:14.328619 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:14.328630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:14.328630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:14.328642 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:14.330939 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:14.331338 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:14.331344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:14.331344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:14.331442 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:14.331683 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:14.331684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:14.331684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:14.331931 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:14.332115 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:14.332204 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 08:18:14.332310 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260620 08:18:14.333603 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:14.349557 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:14.338188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46273 (local address 127.0.0.1:42893)
0620 08:18:14.338228 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:14.338230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:14.342112 (+ 3882us) server_negotiation.cc:407] Connection header received
0620 08:18:14.342207 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:14.342208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:14.342220 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:14.342296 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:14.342867 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:14.343400 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:14.344130 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:14.344529 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:14.344647 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:14.346330 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:14.346332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:14.346332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:14.346339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:14.348110 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:14.348563 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:14.348564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:14.348564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:14.348627 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:14.348878 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:14.348879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:14.348880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:14.349013 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:14.349125 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:14.349386 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 08:18:14.349450 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":20}
I20260620 08:18:14.352443 13394 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:14.353173 13394 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:14.364198 13394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32eb1c48770549f5964f60cff8533200: Generated new TSK 0
I20260620 08:18:14.364399 13394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:14.366629 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13315
W20260620 08:18:14.371403 13300 connection.cc:535] client connection to 127.0.0.1:42893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42893): Connection reset by peer (error 104)
2026-06-20T08:18:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:14 dist-test-slave-65w8 krb5kdc[13414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:14 dist-test-slave-65w8 krb5kdc[13414](info): set up 2 sockets
Jun 20 08:18:14 dist-test-slave-65w8 krb5kdc[13414](info): commencing operation
krb5kdc: starting...
W20260620 08:18:15.434176 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:15 dist-test-slave-65w8 krb5kdc[13414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:15.561023 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:15.642599 13427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:15.642834 13427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:15.642858 13427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:15.644050 13427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:15.644091 13427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:15.644106 13427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:15.644119 13427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:15.644133 13427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:15.645629 13427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:15.645838 13427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:15.646112 13427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:15.651022 13435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:15.651033 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:15.651116 13427 server_base.cc:1034] running on GCE node
W20260620 08:18:15.651039 13433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:15.651422 13427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:15.651732 13427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:15.652921 13427 hybrid_clock.cc:648] HybridClock initialized: now 1781943495652902 us; error 40 us; skew 500 ppm
Jun 20 08:18:15 dist-test-slave-65w8 krb5kdc[13414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:15.661806 13427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:15.662282 13427 webserver.cc:460] Webserver started at http://127.0.0.1:40105/ using document root <none> and password file <none>
I20260620 08:18:15.662536 13427 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:15.662585 13427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:15.662712 13427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:15.663795 13427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "c75a60348eb44f2c9578ce647c417a0b"
format_stamp: "Formatted at 2026-06-20 08:18:15 on dist-test-slave-65w8"
server_key: "36ec946d88c02b24afc39d21ddfa95ed"
server_key_iv: "e93ab6f1d3c541f543910a4be4cc1f85"
server_key_version: "encryptionkey@0"
I20260620 08:18:15.664160 13427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "c75a60348eb44f2c9578ce647c417a0b"
format_stamp: "Formatted at 2026-06-20 08:18:15 on dist-test-slave-65w8"
server_key: "36ec946d88c02b24afc39d21ddfa95ed"
server_key_iv: "e93ab6f1d3c541f543910a4be4cc1f85"
server_key_version: "encryptionkey@0"
I20260620 08:18:15.665611 13427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 08:18:15.666646 13442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:15.666916 13427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:15.667029 13427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "c75a60348eb44f2c9578ce647c417a0b"
format_stamp: "Formatted at 2026-06-20 08:18:15 on dist-test-slave-65w8"
server_key: "36ec946d88c02b24afc39d21ddfa95ed"
server_key_iv: "e93ab6f1d3c541f543910a4be4cc1f85"
server_key_version: "encryptionkey@0"
I20260620 08:18:15.667104 13427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:15.684227 13427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:15.686677 13427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:15.686844 13427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:15.691108 13427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38323
I20260620 08:18:15.691175 13494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38323 every 8 connection(s)
I20260620 08:18:15.691555 13427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:15.692613 13495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:15.695060 13495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b: Bootstrap starting.
I20260620 08:18:15.695664 13495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:15.695967 13495 log.cc:824] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:15.696257 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13427
I20260620 08:18:15.696395 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:15.696600 25574 external_mini_cluster.cc:1428] Setting key 1cc6be47a2ea010e85e9b70bf7d0bfc7
I20260620 08:18:15.696763 13495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b: No bootstrap required, opened a new log
I20260620 08:18:15.698654 13495 raft_consensus.cc:348] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a60348eb44f2c9578ce647c417a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38323 } }
I20260620 08:18:15.698825 13495 raft_consensus.cc:374] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:15.698872 13495 raft_consensus.cc:729] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75a60348eb44f2c9578ce647c417a0b, State: Initialized, Role: FOLLOWER
I20260620 08:18:15.699000 13495 consensus_queue.cc:260] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a60348eb44f2c9578ce647c417a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38323 } }
I20260620 08:18:15.699102 13495 raft_consensus.cc:388] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:15.699148 13495 raft_consensus.cc:482] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:15.699208 13495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:15.699918 13495 raft_consensus.cc:504] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a60348eb44f2c9578ce647c417a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38323 } }
I20260620 08:18:15.700053 13495 leader_election.cc:302] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c75a60348eb44f2c9578ce647c417a0b; no voters:
I20260620 08:18:15.700254 13495 leader_election.cc:288] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:15.700276 13500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:15.700387 13500 raft_consensus.cc:686] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [term 1 LEADER]: Becoming Leader. State: Replica: c75a60348eb44f2c9578ce647c417a0b, State: Running, Role: LEADER
I20260620 08:18:15.700470 13500 consensus_queue.cc:237] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a60348eb44f2c9578ce647c417a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38323 } }
I20260620 08:18:15.700531 13495 sys_catalog.cc:564] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:15.701555 13502 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c75a60348eb44f2c9578ce647c417a0b. Latest consensus state: current_term: 1 leader_uuid: "c75a60348eb44f2c9578ce647c417a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a60348eb44f2c9578ce647c417a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38323 } } }
I20260620 08:18:15.701640 13502 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:15.701704 13501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c75a60348eb44f2c9578ce647c417a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a60348eb44f2c9578ce647c417a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38323 } } }
I20260620 08:18:15.701779 13501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:15.701887 13511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:15.702396 13511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:15.703972 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:15.697179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48433 (local address 127.0.0.1:38323)
0620 08:18:15.697366 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:15.697375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:15.697406 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:18:15.698650 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:15.698658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:15.698712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:15.698867 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:15.699120 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:15.700543 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:15.701410 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:15.702093 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:15.702791 (+ 698us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:15.703049 (+ 258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:15.703052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:15.703052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:15.703300 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:15.703337 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:15.703352 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:15.703453 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:15.703659 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 08:18:15.703692 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260620 08:18:15.704447 13511 catalog_manager.cc:1347] Generated new cluster ID: e5eb75a91b834d498ad916e58ab29fb7
I20260620 08:18:15.704494 13511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:15.706408 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:15.715932 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:15.708045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48409 (local address 127.0.0.1:38323)
0620 08:18:15.708108 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:15.708110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:15.712854 (+ 4744us) server_negotiation.cc:407] Connection header received
0620 08:18:15.712939 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:15.712940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:15.712954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:15.713006 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:15.713165 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:15.713702 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:15.714524 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:15.714770 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:15.714864 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:15.715622 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:15.715623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:15.715623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:15.715629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:15.715646 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:15.715650 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:15.715743 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:15.715810 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:18:15.715823 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:18:15.736037 13511 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:15.736841 13511 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:15.742723 13511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c75a60348eb44f2c9578ce647c417a0b: Generated new TSK 0
I20260620 08:18:15.742909 13511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:15.750694 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13427
W20260620 08:18:15.755332 13412 connection.cc:535] client connection to 127.0.0.1:38323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38323): Connection reset by peer (error 104)
2026-06-20T08:18:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:15 dist-test-slave-65w8 krb5kdc[13526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:15 dist-test-slave-65w8 krb5kdc[13526](info): set up 2 sockets
Jun 20 08:18:15 dist-test-slave-65w8 krb5kdc[13526](info): commencing operation
krb5kdc: starting...
W20260620 08:18:16.794680 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:16 dist-test-slave-65w8 krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:16.929059 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:17.017114 13539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:17.017344 13539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:17.017373 13539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:17.018623 13539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:17.018671 13539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:17.018685 13539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:17.018698 13539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:17.018710 13539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:17.020978 13539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:17.021649 13539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:17.022022 13539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:17.031692 13544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:17.031692 13547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:17.031881 13545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:17.032255 13539 server_base.cc:1034] running on GCE node
I20260620 08:18:17.032761 13539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:17.033272 13539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:17.034504 13539 hybrid_clock.cc:648] HybridClock initialized: now 1781943497034460 us; error 55 us; skew 500 ppm
Jun 20 08:18:17 dist-test-slave-65w8 krb5kdc[13526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:17.037397 13539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:17.037824 13539 webserver.cc:460] Webserver started at http://127.0.0.1:33987/ using document root <none> and password file <none>
I20260620 08:18:17.038151 13539 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:17.038229 13539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:17.038384 13539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:17.039654 13539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "cd6277b2369c4f4babc47edb49dae557"
format_stamp: "Formatted at 2026-06-20 08:18:17 on dist-test-slave-65w8"
server_key: "d2274384a5442533a196b8857d5dea7e"
server_key_iv: "fbf3090e971032219d34d462319474f5"
server_key_version: "encryptionkey@0"
I20260620 08:18:17.040194 13539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "cd6277b2369c4f4babc47edb49dae557"
format_stamp: "Formatted at 2026-06-20 08:18:17 on dist-test-slave-65w8"
server_key: "d2274384a5442533a196b8857d5dea7e"
server_key_iv: "fbf3090e971032219d34d462319474f5"
server_key_version: "encryptionkey@0"
I20260620 08:18:17.041739 13539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:17.042874 13554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:17.043256 13539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:17.043376 13539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "cd6277b2369c4f4babc47edb49dae557"
format_stamp: "Formatted at 2026-06-20 08:18:17 on dist-test-slave-65w8"
server_key: "d2274384a5442533a196b8857d5dea7e"
server_key_iv: "fbf3090e971032219d34d462319474f5"
server_key_version: "encryptionkey@0"
I20260620 08:18:17.043478 13539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:17.048521 13539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:17.051189 13539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:17.051385 13539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:17.058013 13539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38405
I20260620 08:18:17.058099 13606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38405 every 8 connection(s)
I20260620 08:18:17.058476 13539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:17.059592 13607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:17.062327 13607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557: Bootstrap starting.
I20260620 08:18:17.062896 13607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:17.063131 13607 log.cc:824] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:17.064374 13607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557: No bootstrap required, opened a new log
I20260620 08:18:17.065130 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13539
I20260620 08:18:17.065372 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:17.065603 25574 external_mini_cluster.cc:1428] Setting key f80d69ae8f6e0f198bbc92af5777c054
I20260620 08:18:17.069167 13607 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6277b2369c4f4babc47edb49dae557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38405 } }
I20260620 08:18:17.069397 13607 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:17.069479 13607 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6277b2369c4f4babc47edb49dae557, State: Initialized, Role: FOLLOWER
I20260620 08:18:17.069680 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:17.066425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49421 (local address 127.0.0.1:38405)
0620 08:18:17.066629 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:17.066637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:17.066663 (+ 26us) server_negotiation.cc:407] Connection header received
0620 08:18:17.067173 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:17.067181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:17.067211 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:17.067364 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:17.068990 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:17.068994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:17.068994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:17.068998 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:17.069038 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:17.069050 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:17.069133 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:17.069205 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 08:18:17.069252 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260620 08:18:17.069639 13607 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6277b2369c4f4babc47edb49dae557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38405 } }
I20260620 08:18:17.069784 13607 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:17.069820 13607 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:17.069895 13607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:17.070771 13607 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6277b2369c4f4babc47edb49dae557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38405 } }
I20260620 08:18:17.070868 13607 leader_election.cc:302] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd6277b2369c4f4babc47edb49dae557; no voters:
I20260620 08:18:17.071058 13607 leader_election.cc:288] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:17.071081 13612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:17.071280 13612 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [term 1 LEADER]: Becoming Leader. State: Replica: cd6277b2369c4f4babc47edb49dae557, State: Running, Role: LEADER
I20260620 08:18:17.071388 13607 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:17.071391 13612 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6277b2369c4f4babc47edb49dae557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38405 } }
I20260620 08:18:17.071525 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:17.071867 13614 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd6277b2369c4f4babc47edb49dae557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6277b2369c4f4babc47edb49dae557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38405 } } }
I20260620 08:18:17.071964 13614 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:17.072180 13613 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6277b2369c4f4babc47edb49dae557. Latest consensus state: current_term: 1 leader_uuid: "cd6277b2369c4f4babc47edb49dae557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6277b2369c4f4babc47edb49dae557" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38405 } } }
I20260620 08:18:17.072237 13613 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:17.072219 13620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:17.075884 13620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:17.076692 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:17.073927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48945 (local address 127.0.0.1:38405)
0620 08:18:17.073982 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:17.073985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:17.074222 (+ 237us) server_negotiation.cc:407] Connection header received
0620 08:18:17.074251 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:17.074252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:17.074258 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:17.074322 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:17.076448 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:17.076450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:17.076450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:17.076455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:17.076474 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:17.076479 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:17.076549 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:17.076610 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 08:18:17.076624 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:18:17.078284 13620 catalog_manager.cc:1347] Generated new cluster ID: 1799d20d24de4435bf345ccee10400d0
I20260620 08:18:17.078344 13620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:17.091130 13620 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:17.091326 13620 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:17.095824 13620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd6277b2369c4f4babc47edb49dae557: Generated new TSK 0
I20260620 08:18:17.096041 13620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:17.101617 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13539
2026-06-20T08:18:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:17 dist-test-slave-65w8 krb5kdc[13638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:17 dist-test-slave-65w8 krb5kdc[13638](info): set up 2 sockets
Jun 20 08:18:17 dist-test-slave-65w8 krb5kdc[13638](info): commencing operation
krb5kdc: starting...
W20260620 08:18:18.170480 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:18 dist-test-slave-65w8 krb5kdc[13638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:18.296151 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:18.377626 13651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:18.377837 13651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:18.377911 13651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:18.379102 13651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:18.379145 13651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:18.379159 13651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:18.379172 13651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:18.379184 13651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:18.380692 13651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:18.380941 13651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:18.381170 13651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:18.386240 13659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:18.386252 13656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:18.386255 13657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:18.386520 13651 server_base.cc:1034] running on GCE node
I20260620 08:18:18.386770 13651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:18.387087 13651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:18.388266 13651 hybrid_clock.cc:648] HybridClock initialized: now 1781943498388256 us; error 36 us; skew 500 ppm
Jun 20 08:18:18 dist-test-slave-65w8 krb5kdc[13638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:18.390928 13651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:18.391438 13651 webserver.cc:460] Webserver started at http://127.0.0.1:46209/ using document root <none> and password file <none>
I20260620 08:18:18.391664 13651 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:18.391734 13651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:18.391886 13651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:18.392959 13651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "c53f9b36603842009a69b00d2550d9b6"
format_stamp: "Formatted at 2026-06-20 08:18:18 on dist-test-slave-65w8"
server_key: "0419c7bff29d43e75ad6e401f9166e57"
server_key_iv: "26cfcac3a03070271ab3e6b650f38173"
server_key_version: "encryptionkey@0"
I20260620 08:18:18.393335 13651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "c53f9b36603842009a69b00d2550d9b6"
format_stamp: "Formatted at 2026-06-20 08:18:18 on dist-test-slave-65w8"
server_key: "0419c7bff29d43e75ad6e401f9166e57"
server_key_iv: "26cfcac3a03070271ab3e6b650f38173"
server_key_version: "encryptionkey@0"
I20260620 08:18:18.394800 13651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:18:18.395618 13666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:18.395843 13651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 08:18:18.395962 13651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "c53f9b36603842009a69b00d2550d9b6"
format_stamp: "Formatted at 2026-06-20 08:18:18 on dist-test-slave-65w8"
server_key: "0419c7bff29d43e75ad6e401f9166e57"
server_key_iv: "26cfcac3a03070271ab3e6b650f38173"
server_key_version: "encryptionkey@0"
I20260620 08:18:18.396044 13651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:18.423406 13651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:18.425623 13651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:18.425798 13651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:18.430013 13651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40207
I20260620 08:18:18.430076 13718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40207 every 8 connection(s)
I20260620 08:18:18.430436 13651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:18.431205 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13651
I20260620 08:18:18.431331 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:18.431332 13719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:18.431527 25574 external_mini_cluster.cc:1428] Setting key 2e33ed95d8b769cd70fcce2bd33c447d
I20260620 08:18:18.434232 13719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6: Bootstrap starting.
I20260620 08:18:18.434837 13719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:18.435114 13719 log.cc:824] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:18.435781 13719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6: No bootstrap required, opened a new log
Jun 20 08:18:18 dist-test-slave-65w8 krb5kdc[13638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:18:18.437990 13719 raft_consensus.cc:348] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f9b36603842009a69b00d2550d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40207 } }
I20260620 08:18:18.438150 13719 raft_consensus.cc:374] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:18.438189 13719 raft_consensus.cc:729] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53f9b36603842009a69b00d2550d9b6, State: Initialized, Role: FOLLOWER
I20260620 08:18:18.438324 13719 consensus_queue.cc:260] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f9b36603842009a69b00d2550d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40207 } }
I20260620 08:18:18.438436 13719 raft_consensus.cc:388] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:18.438468 13719 raft_consensus.cc:482] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:18.438519 13719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:18.439174 13719 raft_consensus.cc:504] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f9b36603842009a69b00d2550d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40207 } }
I20260620 08:18:18.439282 13719 leader_election.cc:302] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c53f9b36603842009a69b00d2550d9b6; no voters:
I20260620 08:18:18.439492 13719 leader_election.cc:288] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:18.439520 13724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:18.439778 13719 sys_catalog.cc:564] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:18.439924 13724 raft_consensus.cc:686] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [term 1 LEADER]: Becoming Leader. State: Replica: c53f9b36603842009a69b00d2550d9b6, State: Running, Role: LEADER
I20260620 08:18:18.440058 13724 consensus_queue.cc:237] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f9b36603842009a69b00d2550d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40207 } }
I20260620 08:18:18.440398 13724 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c53f9b36603842009a69b00d2550d9b6. Latest consensus state: current_term: 1 leader_uuid: "c53f9b36603842009a69b00d2550d9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f9b36603842009a69b00d2550d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40207 } } }
I20260620 08:18:18.440531 13724 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:18.440435 13725 sys_catalog.cc:455] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c53f9b36603842009a69b00d2550d9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c53f9b36603842009a69b00d2550d9b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40207 } } }
I20260620 08:18:18.441258 13725 sys_catalog.cc:458] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:18.441679 13733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:18.442305 13733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:18.442723 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:18.432020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53089 (local address 127.0.0.1:40207)
0620 08:18:18.432205 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:18.432214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:18.432249 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:18:18.432765 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:18.432772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:18.432816 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:18.433030 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:18.433642 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:18.434626 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:18.435513 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:18.435960 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:18.436087 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:18.438678 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:18.438682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:18.438682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:18.438693 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:18.440787 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:18.441315 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:18.441317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:18.441317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:18.441448 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:18.441701 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:18.441702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:18.441703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:18.442037 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:18.442190 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:18.442375 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 08:18:18.442536 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260620 08:18:18.443759 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:18.444083 13733 catalog_manager.cc:1347] Generated new cluster ID: 6316af2d785642afb7744e35c4ab233b
I20260620 08:18:18.444169 13733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:18.449002 13733 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:18.449594 13733 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:18.453814 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:18.445446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36103 (local address 127.0.0.1:40207)
0620 08:18:18.445493 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:18.445495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:18.445504 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:18.445526 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:18.445527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:18.445536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:18.445600 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:18.446504 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:18.447253 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:18.448256 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:18.448598 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:18.448678 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:18.450789 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:18.450791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:18.450791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:18.450799 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:18.452624 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:18.453016 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:18.453017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:18.453017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:18.453079 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:18.453277 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:18.453278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:18.453278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:18.453419 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:18.453518 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:18.453626 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 08:18:18.453716 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":27}
I20260620 08:18:18.458370 13733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c53f9b36603842009a69b00d2550d9b6: Generated new TSK 0
I20260620 08:18:18.458552 13733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:18.461290 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13651
2026-06-20T08:18:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:18 dist-test-slave-65w8 krb5kdc[13750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:18 dist-test-slave-65w8 krb5kdc[13750](info): set up 2 sockets
Jun 20 08:18:18 dist-test-slave-65w8 krb5kdc[13750](info): commencing operation
krb5kdc: starting...
W20260620 08:18:19.518872 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:19 dist-test-slave-65w8 krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:19.644124 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:19.726050 13763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:19.726259 13763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:19.726284 13763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:19.727474 13763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:19.727519 13763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:19.727532 13763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:19.727545 13763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:19.727557 13763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:19.729074 13763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:19.729383 13763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:19.729628 13763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:19.734431 13768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:19.734596 13763 server_base.cc:1034] running on GCE node
W20260620 08:18:19.734524 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:19.734431 13769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:19.734946 13763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:19.735255 13763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:19.736431 13763 hybrid_clock.cc:648] HybridClock initialized: now 1781943499736412 us; error 39 us; skew 500 ppm
Jun 20 08:18:19 dist-test-slave-65w8 krb5kdc[13750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:19.739087 13763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:19.739498 13763 webserver.cc:460] Webserver started at http://127.0.0.1:43677/ using document root <none> and password file <none>
I20260620 08:18:19.739739 13763 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:19.739787 13763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:19.739928 13763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:19.741029 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "8df6d1c0320b49268858f428783ea806"
format_stamp: "Formatted at 2026-06-20 08:18:19 on dist-test-slave-65w8"
server_key: "1badd67343e3a0bbc5adf66e50be9e3d"
server_key_iv: "acbfcdc9508365ad20aa6551b4a743be"
server_key_version: "encryptionkey@0"
I20260620 08:18:19.741395 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "8df6d1c0320b49268858f428783ea806"
format_stamp: "Formatted at 2026-06-20 08:18:19 on dist-test-slave-65w8"
server_key: "1badd67343e3a0bbc5adf66e50be9e3d"
server_key_iv: "acbfcdc9508365ad20aa6551b4a743be"
server_key_version: "encryptionkey@0"
I20260620 08:18:19.742807 13763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:18:19.743583 13778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:19.743773 13763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 08:18:19.743847 13763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "8df6d1c0320b49268858f428783ea806"
format_stamp: "Formatted at 2026-06-20 08:18:19 on dist-test-slave-65w8"
server_key: "1badd67343e3a0bbc5adf66e50be9e3d"
server_key_iv: "acbfcdc9508365ad20aa6551b4a743be"
server_key_version: "encryptionkey@0"
I20260620 08:18:19.743944 13763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:19.763725 13763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:19.766664 13763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:19.766781 13763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:19.770879 13763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43815
I20260620 08:18:19.770946 13830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43815 every 8 connection(s)
I20260620 08:18:19.771314 13763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:19.772202 13831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:19.774854 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806: Bootstrap starting.
I20260620 08:18:19.775509 13831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:19.775818 13831 log.cc:824] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:19.776715 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806: No bootstrap required, opened a new log
I20260620 08:18:19.778548 13831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df6d1c0320b49268858f428783ea806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43815 } }
I20260620 08:18:19.778710 13831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:19.778735 13831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df6d1c0320b49268858f428783ea806, State: Initialized, Role: FOLLOWER
I20260620 08:18:19.778915 13831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df6d1c0320b49268858f428783ea806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43815 } }
I20260620 08:18:19.778990 13831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:19.779012 13831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:19.779044 13831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:19.779322 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13763
I20260620 08:18:19.779448 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:19.779613 25574 external_mini_cluster.cc:1428] Setting key 3187fc5969c98a91ef87dc447a94b417
I20260620 08:18:19.779917 13831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df6d1c0320b49268858f428783ea806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43815 } }
I20260620 08:18:19.780035 13831 leader_election.cc:302] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df6d1c0320b49268858f428783ea806; no voters:
I20260620 08:18:19.780390 13831 leader_election.cc:288] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:19.780404 13836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:19.780648 13836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [term 1 LEADER]: Becoming Leader. State: Replica: 8df6d1c0320b49268858f428783ea806, State: Running, Role: LEADER
I20260620 08:18:19.780759 13831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:19.780887 13836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df6d1c0320b49268858f428783ea806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43815 } }
I20260620 08:18:19.781561 13838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df6d1c0320b49268858f428783ea806. Latest consensus state: current_term: 1 leader_uuid: "8df6d1c0320b49268858f428783ea806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df6d1c0320b49268858f428783ea806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43815 } } }
I20260620 08:18:19.781726 13838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:19.781976 13845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:19.781560 13837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df6d1c0320b49268858f428783ea806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df6d1c0320b49268858f428783ea806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43815 } } }
I20260620 08:18:19.782217 13837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:19.782549 13845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:19.784425 13845 catalog_manager.cc:1347] Generated new cluster ID: 44c3708bc225419a851a141d12989fda
I20260620 08:18:19.784797 13845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:19.786476 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:19.780194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40313 (local address 127.0.0.1:43815)
0620 08:18:19.780375 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:19.780387 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:19.780453 (+ 66us) server_negotiation.cc:407] Connection header received
0620 08:18:19.780630 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:19.780663 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:19.780732 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:19.780978 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:19.781401 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:19.782882 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:19.783887 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:19.784489 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:19.784598 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:19.785494 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:19.785498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:19.785498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:19.785747 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:19.785791 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:19.785808 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:19.785920 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:19.786163 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 08:18:19.786236 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260620 08:18:19.787444 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:19.794319 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:19.789187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60595 (local address 127.0.0.1:43815)
0620 08:18:19.789229 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:19.789231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:19.790135 (+ 904us) server_negotiation.cc:407] Connection header received
0620 08:18:19.790287 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:19.790288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:19.790301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:19.790333 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:19.790574 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:19.791422 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:19.792701 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:19.793072 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:19.793122 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:19.794002 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:19.794004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:19.794004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:19.794011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:19.794028 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:19.794033 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:19.794064 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:19.794187 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 08:18:19.794222 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:19.796005 13845 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:19.796653 13845 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:19.810899 13845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8df6d1c0320b49268858f428783ea806: Generated new TSK 0
I20260620 08:18:19.811146 13845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:19.814782 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13763
2026-06-20T08:18:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:19 dist-test-slave-65w8 krb5kdc[13862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:19 dist-test-slave-65w8 krb5kdc[13862](info): set up 2 sockets
Jun 20 08:18:19 dist-test-slave-65w8 krb5kdc[13862](info): commencing operation
krb5kdc: starting...
W20260620 08:18:20.858177 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:20 dist-test-slave-65w8 krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:20.985610 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:21.070030 13875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:21.070245 13875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:21.070276 13875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:21.071496 13875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:21.071545 13875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:21.071560 13875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:21.071573 13875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:21.071585 13875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:21.073117 13875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:21.073395 13875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:21.073627 13875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 08:18:21.078845 13875 server_base.cc:1034] running on GCE node
W20260620 08:18:21.078801 13883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:21.078761 13881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:21.078761 13880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:21.079308 13875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:21.079656 13875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:21.080826 13875 hybrid_clock.cc:648] HybridClock initialized: now 1781943501080822 us; error 41 us; skew 500 ppm
Jun 20 08:18:21 dist-test-slave-65w8 krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:21.088207 13875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:21.088711 13875 webserver.cc:460] Webserver started at http://127.0.0.1:34853/ using document root <none> and password file <none>
I20260620 08:18:21.088973 13875 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:21.089046 13875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:21.089193 13875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:21.090368 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "d1b9b7eb3e8a400183d107feca6ef64e"
format_stamp: "Formatted at 2026-06-20 08:18:21 on dist-test-slave-65w8"
server_key: "d223b6c89367563932a8712b4c86b44d"
server_key_iv: "b89ca10125f867e1c1c6e49b53530d9d"
server_key_version: "encryptionkey@0"
I20260620 08:18:21.090770 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "d1b9b7eb3e8a400183d107feca6ef64e"
format_stamp: "Formatted at 2026-06-20 08:18:21 on dist-test-slave-65w8"
server_key: "d223b6c89367563932a8712b4c86b44d"
server_key_iv: "b89ca10125f867e1c1c6e49b53530d9d"
server_key_version: "encryptionkey@0"
I20260620 08:18:21.092132 13875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:21.092921 13890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:21.093108 13875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:21.093226 13875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "d1b9b7eb3e8a400183d107feca6ef64e"
format_stamp: "Formatted at 2026-06-20 08:18:21 on dist-test-slave-65w8"
server_key: "d223b6c89367563932a8712b4c86b44d"
server_key_iv: "b89ca10125f867e1c1c6e49b53530d9d"
server_key_version: "encryptionkey@0"
I20260620 08:18:21.093315 13875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:21.096715 13875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:21.098948 13875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:21.099107 13875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:21.103173 13875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42667
I20260620 08:18:21.103215 13942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42667 every 8 connection(s)
I20260620 08:18:21.103590 13875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:21.104624 13943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:21.106988 13943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e: Bootstrap starting.
I20260620 08:18:21.107565 13943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:21.107865 13943 log.cc:824] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:21.108572 13943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e: No bootstrap required, opened a new log
I20260620 08:18:21.110229 13943 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260620 08:18:21.110365 13943 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:21.110388 13943 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b9b7eb3e8a400183d107feca6ef64e, State: Initialized, Role: FOLLOWER
I20260620 08:18:21.110546 13943 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260620 08:18:21.110607 13943 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:21.110628 13943 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:21.110697 13943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:21.111029 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13875
I20260620 08:18:21.111166 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:21.111302 13943 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260620 08:18:21.111336 25574 external_mini_cluster.cc:1428] Setting key f8099ce2b94d7c1318825b0166ac9e67
I20260620 08:18:21.111389 13943 leader_election.cc:302] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b9b7eb3e8a400183d107feca6ef64e; no voters:
I20260620 08:18:21.111543 13943 leader_election.cc:288] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:21.111577 13946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:21.111696 13946 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [term 1 LEADER]: Becoming Leader. State: Replica: d1b9b7eb3e8a400183d107feca6ef64e, State: Running, Role: LEADER
I20260620 08:18:21.111807 13943 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:21.111830 13946 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260620 08:18:21.112246 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b9b7eb3e8a400183d107feca6ef64e. Latest consensus state: current_term: 1 leader_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } } }
I20260620 08:18:21.112251 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b9b7eb3e8a400183d107feca6ef64e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } } }
I20260620 08:18:21.112455 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:21.112389 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:21.112944 13953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:21.113476 13953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:21.114326 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:21.111987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48161 (local address 127.0.0.1:42667)
0620 08:18:21.112506 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:21.112515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:21.112550 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:18:21.112670 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:21.112678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:21.112708 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:21.112957 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:21.113696 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:21.113699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:21.113700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:21.113704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:21.113749 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:21.113763 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:21.113837 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:21.114084 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 08:18:21.114131 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":77,"threads_started":1}
I20260620 08:18:21.115118 13953 catalog_manager.cc:1347] Generated new cluster ID: 1cbdfe06a2964687a2790efc00e8edca
I20260620 08:18:21.115170 13953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:21.115289 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:21.119462 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:21.118239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44353 (local address 127.0.0.1:42667)
0620 08:18:21.118289 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:21.118291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:21.118446 (+ 155us) server_negotiation.cc:407] Connection header received
0620 08:18:21.118523 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:21.118524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:21.118530 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:21.118587 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:21.119217 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:21.119218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:21.119218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:21.119222 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:21.119236 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:21.119240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:21.119288 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:21.119345 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 08:18:21.119358 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:18:21.123756 13953 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:21.123960 13953 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:21.132778 13953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1b9b7eb3e8a400183d107feca6ef64e: Generated new TSK 0
I20260620 08:18:21.132984 13953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:21.136404 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13875
2026-06-20T08:18:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:21 dist-test-slave-65w8 krb5kdc[13974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:21 dist-test-slave-65w8 krb5kdc[13974](info): set up 2 sockets
Jun 20 08:18:21 dist-test-slave-65w8 krb5kdc[13974](info): commencing operation
krb5kdc: starting...
W20260620 08:18:22.178125 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:22 dist-test-slave-65w8 krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:22.302100 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:22.388185 13987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:22.388485 13987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:22.388520 13987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:22.390117 13987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:22.390163 13987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:22.390177 13987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:22.390192 13987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:22.390203 13987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:22.391676 13987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:22.391871 13987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:22.392045 13987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:22.397071 13992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:22.397205 13987 server_base.cc:1034] running on GCE node
W20260620 08:18:22.397058 13995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:22.397102 13993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:22.397639 13987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:22.398043 13987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:22.399297 13987 hybrid_clock.cc:648] HybridClock initialized: now 1781943502399283 us; error 44 us; skew 500 ppm
Jun 20 08:18:22 dist-test-slave-65w8 krb5kdc[13974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:22.402128 13987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:22.402781 13987 webserver.cc:460] Webserver started at http://127.0.0.1:38983/ using document root <none> and password file <none>
I20260620 08:18:22.403095 13987 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:22.403178 13987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:22.403342 13987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:22.405077 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "ab5ae63df3b7492c9e2dcf4a8071b991"
format_stamp: "Formatted at 2026-06-20 08:18:22 on dist-test-slave-65w8"
server_key: "ff11ca8429f6dc466d91a8ade8bc9883"
server_key_iv: "a51e67dd662629da6456ce95c90ca14a"
server_key_version: "encryptionkey@0"
I20260620 08:18:22.405588 13987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "ab5ae63df3b7492c9e2dcf4a8071b991"
format_stamp: "Formatted at 2026-06-20 08:18:22 on dist-test-slave-65w8"
server_key: "ff11ca8429f6dc466d91a8ade8bc9883"
server_key_iv: "a51e67dd662629da6456ce95c90ca14a"
server_key_version: "encryptionkey@0"
I20260620 08:18:22.407512 13987 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 08:18:22.408509 14002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:22.408715 13987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:22.408818 13987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "ab5ae63df3b7492c9e2dcf4a8071b991"
format_stamp: "Formatted at 2026-06-20 08:18:22 on dist-test-slave-65w8"
server_key: "ff11ca8429f6dc466d91a8ade8bc9883"
server_key_iv: "a51e67dd662629da6456ce95c90ca14a"
server_key_version: "encryptionkey@0"
I20260620 08:18:22.408931 13987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:22.434432 13987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:22.436703 13987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:22.436887 13987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:22.441331 13987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40585
I20260620 08:18:22.441392 14054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40585 every 8 connection(s)
I20260620 08:18:22.441777 13987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:22.442732 14055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:22.445122 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991: Bootstrap starting.
I20260620 08:18:22.445745 14055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:22.446094 14055 log.cc:824] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:22.446874 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991: No bootstrap required, opened a new log
I20260620 08:18:22.447737 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 13987
I20260620 08:18:22.447912 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:22.448086 25574 external_mini_cluster.cc:1428] Setting key d53be0ae03dcf66c47bb8287c296b2a9
I20260620 08:18:22.448997 14055 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260620 08:18:22.449162 14055 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:22.449206 14055 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5ae63df3b7492c9e2dcf4a8071b991, State: Initialized, Role: FOLLOWER
I20260620 08:18:22.449363 14055 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260620 08:18:22.449477 14055 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:22.449540 14055 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:22.449597 14055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:22.450659 14055 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260620 08:18:22.450785 14055 leader_election.cc:302] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab5ae63df3b7492c9e2dcf4a8071b991; no voters:
I20260620 08:18:22.451004 14055 leader_election.cc:288] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:22.451038 14060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:22.451149 14060 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [term 1 LEADER]: Becoming Leader. State: Replica: ab5ae63df3b7492c9e2dcf4a8071b991, State: Running, Role: LEADER
I20260620 08:18:22.451294 14055 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:22.451295 14060 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260620 08:18:22.453213 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } } }
I20260620 08:18:22.453250 14062 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab5ae63df3b7492c9e2dcf4a8071b991. Latest consensus state: current_term: 1 leader_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5ae63df3b7492c9e2dcf4a8071b991" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } } }
I20260620 08:18:22.453390 14062 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:22.453601 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:22.454017 14075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:22.454566 14075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 08:18:22 dist-test-slave-65w8 krb5kdc[13974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:18:22.456372 14075 catalog_manager.cc:1347] Generated new cluster ID: 7c95a87a072b4790a172c06d3a171865
I20260620 08:18:22.456423 14075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:22.460398 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:22.448659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:50195 (local address 127.0.0.1:40585)
0620 08:18:22.448995 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:22.449003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:22.449032 (+ 29us) server_negotiation.cc:407] Connection header received
0620 08:18:22.449131 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:22.449136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:22.449180 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:22.449435 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:22.450225 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:22.451352 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:22.452556 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:22.453195 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:22.453303 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:22.456453 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:22.456457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:22.456457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:22.456470 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:22.458611 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:22.459074 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:22.459077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:22.459077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:22.459212 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:22.459449 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:22.459450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:22.459450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:22.459765 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:22.459963 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:22.460048 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 08:18:22.460156 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
I20260620 08:18:22.461513 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:22.470712 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:22.463167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:35419 (local address 127.0.0.1:40585)
0620 08:18:22.463215 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:22.463217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:22.463230 (+ 13us) server_negotiation.cc:407] Connection header received
0620 08:18:22.463312 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:22.463313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:22.463322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:22.463367 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:22.464040 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:22.464769 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:22.465844 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:22.466176 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:22.466237 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:22.467725 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:22.467726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:22.467726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:22.467734 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:22.469332 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:22.469768 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:22.469769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:22.469770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:22.469832 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:22.470078 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:22.470079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:22.470079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:22.470203 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:22.470294 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:22.470456 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 08:18:22.470521 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 08:18:22.495360 14075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:22.496217 14075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:22.499678 14075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab5ae63df3b7492c9e2dcf4a8071b991: Generated new TSK 0
I20260620 08:18:22.499907 14075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:22.509266 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 13987
W20260620 08:18:22.513832 13972 connection.cc:535] client connection to 127.0.0.1:40585 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40585): Connection reset by peer (error 104)
2026-06-20T08:18:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:22 dist-test-slave-65w8 krb5kdc[14086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:22 dist-test-slave-65w8 krb5kdc[14086](info): set up 2 sockets
Jun 20 08:18:22 dist-test-slave-65w8 krb5kdc[14086](info): commencing operation
krb5kdc: starting...
W20260620 08:18:23.582273 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:23 dist-test-slave-65w8 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:23.708500 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:23.790930 14099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:23.791147 14099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:23.791172 14099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:23.792403 14099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:23.792452 14099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:23.792466 14099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:23.792479 14099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:23.792491 14099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:23.794056 14099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:23.794325 14099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:23.794577 14099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:23.799994 14105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:23.800112 14104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:23.800004 14107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:23.800287 14099 server_base.cc:1034] running on GCE node
I20260620 08:18:23.800524 14099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:23.800838 14099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:23.802039 14099 hybrid_clock.cc:648] HybridClock initialized: now 1781943503802027 us; error 39 us; skew 500 ppm
Jun 20 08:18:23 dist-test-slave-65w8 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:23.804876 14099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:23.805449 14099 webserver.cc:460] Webserver started at http://127.0.0.1:35427/ using document root <none> and password file <none>
I20260620 08:18:23.805689 14099 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:23.805760 14099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:23.805944 14099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:23.807050 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "7489774f4e294bd6993d30e792b1cead"
format_stamp: "Formatted at 2026-06-20 08:18:23 on dist-test-slave-65w8"
server_key: "4425b30c2ca7890b39f27c7a2c77d802"
server_key_iv: "4cd483462431a442132213dc23d3cd57"
server_key_version: "encryptionkey@0"
I20260620 08:18:23.807435 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "7489774f4e294bd6993d30e792b1cead"
format_stamp: "Formatted at 2026-06-20 08:18:23 on dist-test-slave-65w8"
server_key: "4425b30c2ca7890b39f27c7a2c77d802"
server_key_iv: "4cd483462431a442132213dc23d3cd57"
server_key_version: "encryptionkey@0"
I20260620 08:18:23.808848 14099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:23.809717 14114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:23.810026 14099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:23.810139 14099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "7489774f4e294bd6993d30e792b1cead"
format_stamp: "Formatted at 2026-06-20 08:18:23 on dist-test-slave-65w8"
server_key: "4425b30c2ca7890b39f27c7a2c77d802"
server_key_iv: "4cd483462431a442132213dc23d3cd57"
server_key_version: "encryptionkey@0"
I20260620 08:18:23.810230 14099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:23.832733 14099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:23.835067 14099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:23.835239 14099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:23.839843 14099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35253
I20260620 08:18:23.839922 14166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35253 every 8 connection(s)
I20260620 08:18:23.840284 14099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:23.841147 14167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:23.843590 14167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead: Bootstrap starting.
I20260620 08:18:23.844141 14167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:23.844285 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14099
I20260620 08:18:23.844429 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:23.844512 14167 log.cc:824] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:23.844647 25574 external_mini_cluster.cc:1428] Setting key 6e0f9926068da32113d85650065df228
I20260620 08:18:23.845386 14167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead: No bootstrap required, opened a new log
I20260620 08:18:23.847662 14167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7489774f4e294bd6993d30e792b1cead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 08:18:23.847846 14167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:23.847896 14167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7489774f4e294bd6993d30e792b1cead, State: Initialized, Role: FOLLOWER
I20260620 08:18:23.848027 14167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7489774f4e294bd6993d30e792b1cead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 08:18:23.848110 14167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:23.848171 14167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:23.848233 14167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:23.848958 14167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7489774f4e294bd6993d30e792b1cead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 08:18:23.849098 14167 leader_election.cc:302] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7489774f4e294bd6993d30e792b1cead; no voters:
I20260620 08:18:23.849282 14167 leader_election.cc:288] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:23.849449 14172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:23.849538 14167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:23.849655 14172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [term 1 LEADER]: Becoming Leader. State: Replica: 7489774f4e294bd6993d30e792b1cead, State: Running, Role: LEADER
I20260620 08:18:23.849818 14172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7489774f4e294bd6993d30e792b1cead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260620 08:18:23.850296 14174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7489774f4e294bd6993d30e792b1cead. Latest consensus state: current_term: 1 leader_uuid: "7489774f4e294bd6993d30e792b1cead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7489774f4e294bd6993d30e792b1cead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } } }
I20260620 08:18:23.850348 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7489774f4e294bd6993d30e792b1cead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7489774f4e294bd6993d30e792b1cead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } } }
I20260620 08:18:23.850463 14174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:23.850489 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:23.845197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:53271 (local address 127.0.0.1:35253)
0620 08:18:23.845406 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:23.845415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:23.845449 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:18:23.845937 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:23.845944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:23.845994 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:23.846147 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:23.846376 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:23.847258 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:23.848159 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:23.848657 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:23.848778 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:23.849632 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:23.849636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:23.849637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:23.849851 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:23.849916 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:23.849931 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:23.850030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:23.850128 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 08:18:23.850161 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260620 08:18:23.850474 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:23.850713 14181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:23.851190 14181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:23.851605 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:23.853013 14181 catalog_manager.cc:1347] Generated new cluster ID: 0e49ddfaaa0d45ea85d90a08ac292258
I20260620 08:18:23.853068 14181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:23.856868 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:23.853289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:34231 (local address 127.0.0.1:35253)
0620 08:18:23.853322 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:23.853324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:23.853332 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:23.853391 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:23.853391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:23.853401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:23.853466 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:23.853599 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:23.854331 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:23.855471 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:23.855791 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:23.855869 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:23.856612 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:23.856613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:23.856614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:23.856619 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:23.856636 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:23.856640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:23.856707 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:23.856780 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 08:18:23.856791 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 08:18:23.860455 14181 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:23.861198 14181 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:23.869426 14181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7489774f4e294bd6993d30e792b1cead: Generated new TSK 0
I20260620 08:18:23.869630 14181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:23.874755 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14099
W20260620 08:18:23.879221 14084 connection.cc:535] client connection to 127.0.0.1:35253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35253): Connection reset by peer (error 104)
2026-06-20T08:18:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:23 dist-test-slave-65w8 krb5kdc[14198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:23 dist-test-slave-65w8 krb5kdc[14198](info): set up 2 sockets
Jun 20 08:18:23 dist-test-slave-65w8 krb5kdc[14198](info): commencing operation
krb5kdc: starting...
W20260620 08:18:24.926182 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:25 dist-test-slave-65w8 krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:25.050124 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:25.131309 14211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:25.131582 14211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:25.131624 14211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:25.132817 14211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:25.132897 14211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:25.132921 14211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:25.132942 14211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:25.132970 14211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:25.134546 14211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:25.134845 14211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:25.135082 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:25.140290 14217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:25.140303 14216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:25.140667 14211 server_base.cc:1034] running on GCE node
W20260620 08:18:25.140303 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:25.141074 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:25.141419 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:25.142616 14211 hybrid_clock.cc:648] HybridClock initialized: now 1781943505142603 us; error 31 us; skew 500 ppm
Jun 20 08:18:25 dist-test-slave-65w8 krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:25.145190 14211 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:25.145663 14211 webserver.cc:460] Webserver started at http://127.0.0.1:37487/ using document root <none> and password file <none>
I20260620 08:18:25.145967 14211 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:25.146015 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:25.146169 14211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:25.147306 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "c4bbd6e141004c62894a70c89acf95cd"
format_stamp: "Formatted at 2026-06-20 08:18:25 on dist-test-slave-65w8"
server_key: "00a7c6b681ec11053b03c01daf1a33fe"
server_key_iv: "06a8b83dbf86f6387d55bb1b45570bae"
server_key_version: "encryptionkey@0"
I20260620 08:18:25.147677 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "c4bbd6e141004c62894a70c89acf95cd"
format_stamp: "Formatted at 2026-06-20 08:18:25 on dist-test-slave-65w8"
server_key: "00a7c6b681ec11053b03c01daf1a33fe"
server_key_iv: "06a8b83dbf86f6387d55bb1b45570bae"
server_key_version: "encryptionkey@0"
I20260620 08:18:25.149039 14211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:25.149806 14226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:25.150000 14211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:18:25.150103 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "c4bbd6e141004c62894a70c89acf95cd"
format_stamp: "Formatted at 2026-06-20 08:18:25 on dist-test-slave-65w8"
server_key: "00a7c6b681ec11053b03c01daf1a33fe"
server_key_iv: "06a8b83dbf86f6387d55bb1b45570bae"
server_key_version: "encryptionkey@0"
I20260620 08:18:25.150187 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:25.153707 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:25.156028 14211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:25.156184 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:25.160210 14211 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41885
I20260620 08:18:25.160241 14278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41885 every 8 connection(s)
I20260620 08:18:25.160567 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:25.161616 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:25.163940 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd: Bootstrap starting.
I20260620 08:18:25.164534 14279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:25.164820 14279 log.cc:824] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:25.165304 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14211
I20260620 08:18:25.165480 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:25.165663 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd: No bootstrap required, opened a new log
I20260620 08:18:25.165689 25574 external_mini_cluster.cc:1428] Setting key 2a8dec9cabc63b2f1129ea37853019d4
I20260620 08:18:25.168025 14279 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bbd6e141004c62894a70c89acf95cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260620 08:18:25.168265 14279 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:25.168318 14279 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4bbd6e141004c62894a70c89acf95cd, State: Initialized, Role: FOLLOWER
I20260620 08:18:25.168480 14279 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bbd6e141004c62894a70c89acf95cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260620 08:18:25.168618 14279 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:25.168663 14279 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:25.168711 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:25.166244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:48227 (local address 127.0.0.1:41885)
0620 08:18:25.166458 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:25.166468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:25.166506 (+ 38us) server_negotiation.cc:407] Connection header received
0620 08:18:25.167004 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:25.167010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:25.167035 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:25.167187 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:25.167927 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:25.167931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:25.167931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:25.167936 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:25.167988 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:25.167999 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:25.168078 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:25.168147 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 08:18:25.168195 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260620 08:18:25.168715 14279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:25.169466 14279 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bbd6e141004c62894a70c89acf95cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260620 08:18:25.169588 14279 leader_election.cc:302] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4bbd6e141004c62894a70c89acf95cd; no voters:
I20260620 08:18:25.169636 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:25.169751 14279 leader_election.cc:288] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:25.169812 14284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:25.170011 14279 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:25.170009 14284 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [term 1 LEADER]: Becoming Leader. State: Replica: c4bbd6e141004c62894a70c89acf95cd, State: Running, Role: LEADER
I20260620 08:18:25.170138 14284 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bbd6e141004c62894a70c89acf95cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260620 08:18:25.170553 14285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4bbd6e141004c62894a70c89acf95cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bbd6e141004c62894a70c89acf95cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } } }
I20260620 08:18:25.170544 14284 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4bbd6e141004c62894a70c89acf95cd. Latest consensus state: current_term: 1 leader_uuid: "c4bbd6e141004c62894a70c89acf95cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bbd6e141004c62894a70c89acf95cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } } }
I20260620 08:18:25.170645 14285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:25.170650 14284 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:25.171622 14293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:25.172539 14293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:25.172796 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:25.171292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:41133 (local address 127.0.0.1:41885)
0620 08:18:25.171328 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:25.171330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:25.171518 (+ 188us) server_negotiation.cc:407] Connection header received
0620 08:18:25.171682 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:25.171683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:25.171689 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:25.171750 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:25.172550 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:25.172551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:25.172551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:25.172556 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:25.172570 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:25.172575 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:25.172634 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:25.172722 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 08:18:25.172738 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 08:18:25.174180 14293 catalog_manager.cc:1347] Generated new cluster ID: 36685433eea94507a199e69f9fcb73cf
I20260620 08:18:25.174234 14293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:25.186913 14293 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:25.187115 14293 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:25.198956 14293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4bbd6e141004c62894a70c89acf95cd: Generated new TSK 0
I20260620 08:18:25.199169 14293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:25.203575 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14211
2026-06-20T08:18:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:25 dist-test-slave-65w8 krb5kdc[14310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:25 dist-test-slave-65w8 krb5kdc[14310](info): set up 2 sockets
Jun 20 08:18:25 dist-test-slave-65w8 krb5kdc[14310](info): commencing operation
krb5kdc: starting...
W20260620 08:18:26.279062 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:26 dist-test-slave-65w8 krb5kdc[14310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:26.408631 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:26.491967 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:26.492199 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:26.492226 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:26.493518 14323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:26.493562 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:26.493577 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:26.493588 14323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:26.493600 14323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:26.495167 14323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:26.495432 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:26.495668 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:26.500764 14328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:26.500764 14331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:26.500793 14329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:26.500780 14323 server_base.cc:1034] running on GCE node
I20260620 08:18:26.501284 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:26.501600 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:26.502770 14323 hybrid_clock.cc:648] HybridClock initialized: now 1781943506502756 us; error 44 us; skew 500 ppm
Jun 20 08:18:26 dist-test-slave-65w8 krb5kdc[14310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:26.509977 14323 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:26.510483 14323 webserver.cc:460] Webserver started at http://127.0.0.1:43735/ using document root <none> and password file <none>
I20260620 08:18:26.510746 14323 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:26.510816 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:26.510963 14323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:26.512030 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "a1b1059f7a014b9c96499bd4c8573af4"
format_stamp: "Formatted at 2026-06-20 08:18:26 on dist-test-slave-65w8"
server_key: "649d1e961c8a133a9e905dec3ae942b6"
server_key_iv: "3b490b4f644a78a4049626d0c9a3ccd7"
server_key_version: "encryptionkey@0"
I20260620 08:18:26.512391 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "a1b1059f7a014b9c96499bd4c8573af4"
format_stamp: "Formatted at 2026-06-20 08:18:26 on dist-test-slave-65w8"
server_key: "649d1e961c8a133a9e905dec3ae942b6"
server_key_iv: "3b490b4f644a78a4049626d0c9a3ccd7"
server_key_version: "encryptionkey@0"
I20260620 08:18:26.513751 14323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:26.514557 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:26.514882 14323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:26.514992 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "a1b1059f7a014b9c96499bd4c8573af4"
format_stamp: "Formatted at 2026-06-20 08:18:26 on dist-test-slave-65w8"
server_key: "649d1e961c8a133a9e905dec3ae942b6"
server_key_iv: "3b490b4f644a78a4049626d0c9a3ccd7"
server_key_version: "encryptionkey@0"
I20260620 08:18:26.515064 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:26.527667 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:26.529583 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:26.529752 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:26.533890 14323 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41549
I20260620 08:18:26.533941 14390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41549 every 8 connection(s)
I20260620 08:18:26.534299 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:26.535130 14391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:26.537365 14391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4: Bootstrap starting.
I20260620 08:18:26.537981 14391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:26.538233 14391 log.cc:824] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:26.538964 14391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4: No bootstrap required, opened a new log
I20260620 08:18:26.540764 14391 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b1059f7a014b9c96499bd4c8573af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260620 08:18:26.540918 14391 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:26.540943 14391 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b1059f7a014b9c96499bd4c8573af4, State: Initialized, Role: FOLLOWER
I20260620 08:18:26.541074 14391 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b1059f7a014b9c96499bd4c8573af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260620 08:18:26.541150 14391 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:26.541172 14391 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:26.541200 14391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:26.541710 14391 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b1059f7a014b9c96499bd4c8573af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260620 08:18:26.541795 14391 leader_election.cc:302] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b1059f7a014b9c96499bd4c8573af4; no voters:
I20260620 08:18:26.541996 14391 leader_election.cc:288] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:26.542111 14394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:26.542246 14391 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:26.542300 14394 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [term 1 LEADER]: Becoming Leader. State: Replica: a1b1059f7a014b9c96499bd4c8573af4, State: Running, Role: LEADER
I20260620 08:18:26.542421 14394 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b1059f7a014b9c96499bd4c8573af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } }
I20260620 08:18:26.542894 14396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b1059f7a014b9c96499bd4c8573af4. Latest consensus state: current_term: 1 leader_uuid: "a1b1059f7a014b9c96499bd4c8573af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b1059f7a014b9c96499bd4c8573af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } } }
I20260620 08:18:26.542999 14396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:26.543030 14395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b1059f7a014b9c96499bd4c8573af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b1059f7a014b9c96499bd4c8573af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41549 } } }
I20260620 08:18:26.543092 14395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:26.543313 14403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:26.543895 14403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:26.544196 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14323
I20260620 08:18:26.544332 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:26.544512 25574 external_mini_cluster.cc:1428] Setting key 4eb734bc36a03910b4ba77c610c3689c
I20260620 08:18:26.545917 14403 catalog_manager.cc:1347] Generated new cluster ID: bb857a11a65d46b99a72082eccee4b99
I20260620 08:18:26.545969 14403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:18:26 dist-test-slave-65w8 krb5kdc[14310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 08:18:26.555547 14403 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:26.556154 14403 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:26.557288 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:26.545017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:56903 (local address 127.0.0.1:41549)
0620 08:18:26.545175 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:26.545185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:26.545266 (+ 81us) server_negotiation.cc:407] Connection header received
0620 08:18:26.545515 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:26.545521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:26.545566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:26.545707 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:26.546479 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:26.548112 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:26.549196 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:26.549956 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:26.550096 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:26.553161 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:26.553166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:26.553167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:26.553180 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:26.555304 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:26.555955 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:26.555958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:26.555958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:26.556066 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:26.556347 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:26.556348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:26.556348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:26.556621 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:26.556779 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:26.556952 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 08:18:26.557052 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 08:18:26.558712 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:26.560702 14403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b1059f7a014b9c96499bd4c8573af4: Generated new TSK 0
I20260620 08:18:26.560868 14403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:26.566884 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:26.560499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:58091 (local address 127.0.0.1:41549)
0620 08:18:26.560533 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:26.560535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:26.560554 (+ 19us) server_negotiation.cc:407] Connection header received
0620 08:18:26.560573 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:26.560574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:26.560582 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:26.560627 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:26.561294 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:26.561815 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:26.562575 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:26.562808 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:26.562885 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:26.564195 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:26.564197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:26.564197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:26.564204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:26.565590 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:26.566063 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:26.566064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:26.566064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:26.566122 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:26.566352 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:26.566353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:26.566353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:26.566465 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:26.566590 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:26.566707 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 08:18:26.566784 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:18:26.568724 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14323
W20260620 08:18:26.573443 14308 connection.cc:535] client connection to 127.0.0.1:41549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41549): Connection reset by peer (error 104)
2026-06-20T08:18:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:26 dist-test-slave-65w8 krb5kdc[14422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:26 dist-test-slave-65w8 krb5kdc[14422](info): set up 2 sockets
Jun 20 08:18:26 dist-test-slave-65w8 krb5kdc[14422](info): commencing operation
krb5kdc: starting...
W20260620 08:18:27.630276 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:27 dist-test-slave-65w8 krb5kdc[14422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:27.755174 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:27.836849 14435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:27.837097 14435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:27.837134 14435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:27.838311 14435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:27.838380 14435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:27.838405 14435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:27.838431 14435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:27.838464 14435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:27.839956 14435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:27.840240 14435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:27.840464 14435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:27.845709 14440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:27.845731 14441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:27.845933 14435 server_base.cc:1034] running on GCE node
W20260620 08:18:27.845723 14443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:27.846220 14435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:27.846489 14435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:27.847642 14435 hybrid_clock.cc:648] HybridClock initialized: now 1781943507847629 us; error 28 us; skew 500 ppm
Jun 20 08:18:27 dist-test-slave-65w8 krb5kdc[14422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:27.850227 14435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:27.850693 14435 webserver.cc:460] Webserver started at http://127.0.0.1:33289/ using document root <none> and password file <none>
I20260620 08:18:27.850944 14435 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:27.851012 14435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:27.851161 14435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:27.852218 14435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "4e1f0602b4e943f8b5bb13ebda1709d9"
format_stamp: "Formatted at 2026-06-20 08:18:27 on dist-test-slave-65w8"
server_key: "765a8b16aeb98b4ac57f761609aad7f8"
server_key_iv: "c640201b9af98f9669b1da32f3a936a1"
server_key_version: "encryptionkey@0"
I20260620 08:18:27.852595 14435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "4e1f0602b4e943f8b5bb13ebda1709d9"
format_stamp: "Formatted at 2026-06-20 08:18:27 on dist-test-slave-65w8"
server_key: "765a8b16aeb98b4ac57f761609aad7f8"
server_key_iv: "c640201b9af98f9669b1da32f3a936a1"
server_key_version: "encryptionkey@0"
I20260620 08:18:27.854005 14435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:18:27.854848 14450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:27.855067 14435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:27.855175 14435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "4e1f0602b4e943f8b5bb13ebda1709d9"
format_stamp: "Formatted at 2026-06-20 08:18:27 on dist-test-slave-65w8"
server_key: "765a8b16aeb98b4ac57f761609aad7f8"
server_key_iv: "c640201b9af98f9669b1da32f3a936a1"
server_key_version: "encryptionkey@0"
I20260620 08:18:27.855293 14435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:27.870180 14435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:27.872236 14435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:27.872407 14435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:27.876541 14435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42383
I20260620 08:18:27.876577 14502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42383 every 8 connection(s)
I20260620 08:18:27.876966 14435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:27.877913 14503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:27.880077 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14435
I20260620 08:18:27.880234 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:27.880182 14503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9: Bootstrap starting.
I20260620 08:18:27.880414 25574 external_mini_cluster.cc:1428] Setting key 5c70a13c8493a160ef555c3c2380fdd2
I20260620 08:18:27.880903 14503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:27.881238 14503 log.cc:824] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:27.881997 14503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9: No bootstrap required, opened a new log
I20260620 08:18:27.884096 14503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260620 08:18:27.884271 14503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:27.884308 14503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1f0602b4e943f8b5bb13ebda1709d9, State: Initialized, Role: FOLLOWER
I20260620 08:18:27.884440 14503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260620 08:18:27.884508 14503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:27.884558 14503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:27.884608 14503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:27.885283 14503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260620 08:18:27.885373 14503 leader_election.cc:302] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1f0602b4e943f8b5bb13ebda1709d9; no voters:
I20260620 08:18:27.885557 14503 leader_election.cc:288] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:27.885577 14508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:27.885680 14508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [term 1 LEADER]: Becoming Leader. State: Replica: 4e1f0602b4e943f8b5bb13ebda1709d9, State: Running, Role: LEADER
I20260620 08:18:27.885769 14508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } }
I20260620 08:18:27.885818 14503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:27.886132 14508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } } }
I20260620 08:18:27.886209 14508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:27.886364 14508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1f0602b4e943f8b5bb13ebda1709d9. Latest consensus state: current_term: 1 leader_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1f0602b4e943f8b5bb13ebda1709d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42383 } } }
I20260620 08:18:27.886439 14508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:27.886601 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:27.880969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:60591 (local address 127.0.0.1:42383)
0620 08:18:27.881150 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:27.881158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:27.881185 (+ 27us) server_negotiation.cc:407] Connection header received
0620 08:18:27.881699 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:27.881706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:27.881766 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:27.881943 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:27.882171 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:27.883062 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:27.884246 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:27.884853 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:27.884935 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:27.885732 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:27.885737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:27.885737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:27.886015 (+ 278us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:27.886047 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:27.886059 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:27.886310 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:27.886357 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 08:18:27.886385 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260620 08:18:27.887634 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 08:18:27.887765 14524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:27.887849 14524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:27.887928 14514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:27.888098 14514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:27.889582 14514 catalog_manager.cc:1347] Generated new cluster ID: b5520aba92d84abb988aaaced45dde82
I20260620 08:18:27.889633 14514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:27.893105 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:27.889185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:47161 (local address 127.0.0.1:42383)
0620 08:18:27.889221 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:27.889222 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:27.889335 (+ 113us) server_negotiation.cc:407] Connection header received
0620 08:18:27.889444 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:27.889445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:27.889456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:27.889484 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:27.889679 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:27.890445 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:27.891618 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:27.891935 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:27.892003 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:27.892827 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:27.892829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:27.892829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:27.892835 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:27.892851 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:27.892856 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:27.892905 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:27.893017 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:18:27.893028 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:18:27.906407 14514 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:27.906962 14514 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:27.912226 14514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e1f0602b4e943f8b5bb13ebda1709d9: Generated new TSK 0
I20260620 08:18:27.912448 14514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:27.923059 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14435
W20260620 08:18:27.927759 14420 connection.cc:535] client connection to 127.0.0.1:42383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42383): Connection reset by peer (error 104)
2026-06-20T08:18:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:27 dist-test-slave-65w8 krb5kdc[14534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:27 dist-test-slave-65w8 krb5kdc[14534](info): set up 2 sockets
Jun 20 08:18:27 dist-test-slave-65w8 krb5kdc[14534](info): commencing operation
krb5kdc: starting...
W20260620 08:18:28.974238 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:29 dist-test-slave-65w8 krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 08:18:29.097222 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:29.179946 14547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:29.180209 14547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:29.180250 14547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:29.181427 14547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:29.181507 14547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:29.181532 14547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:29.181562 14547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:29.181594 14547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:29.183125 14547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:29.183422 14547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:29.183652 14547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:29.188797 14552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:29.188792 14555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:29.188792 14553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:29.189263 14547 server_base.cc:1034] running on GCE node
I20260620 08:18:29.189504 14547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:29.189805 14547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:29.190995 14547 hybrid_clock.cc:648] HybridClock initialized: now 1781943509190975 us; error 42 us; skew 500 ppm
Jun 20 08:18:29 dist-test-slave-65w8 krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:29.193570 14547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 08:18:29.194064 14547 webserver.cc:460] Webserver started at http://127.0.0.1:32779/ using document root <none> and password file <none>
I20260620 08:18:29.194288 14547 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:29.194355 14547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:29.194509 14547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:29.195583 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "5387132bc2304ea6bfa887ae12a5ddca"
format_stamp: "Formatted at 2026-06-20 08:18:29 on dist-test-slave-65w8"
server_key: "4c9890ea8dc8bb560e17ad52399c8ee5"
server_key_iv: "6dae6c629274cba191aa142427eb8e43"
server_key_version: "encryptionkey@0"
I20260620 08:18:29.195952 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "5387132bc2304ea6bfa887ae12a5ddca"
format_stamp: "Formatted at 2026-06-20 08:18:29 on dist-test-slave-65w8"
server_key: "4c9890ea8dc8bb560e17ad52399c8ee5"
server_key_iv: "6dae6c629274cba191aa142427eb8e43"
server_key_version: "encryptionkey@0"
I20260620 08:18:29.197316 14547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:18:29.198303 14562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:29.198570 14547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:29.198705 14547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "5387132bc2304ea6bfa887ae12a5ddca"
format_stamp: "Formatted at 2026-06-20 08:18:29 on dist-test-slave-65w8"
server_key: "4c9890ea8dc8bb560e17ad52399c8ee5"
server_key_iv: "6dae6c629274cba191aa142427eb8e43"
server_key_version: "encryptionkey@0"
I20260620 08:18:29.198808 14547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:29.202420 14547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:29.204664 14547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:29.204866 14547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:29.209060 14547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34009
I20260620 08:18:29.209091 14614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34009 every 8 connection(s)
I20260620 08:18:29.209460 14547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:29.210541 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:29.212294 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14547
I20260620 08:18:29.212433 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:29.212623 25574 external_mini_cluster.cc:1428] Setting key 66b2bac0a7e2917c243d877813b6a4cf
I20260620 08:18:29.212975 14615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca: Bootstrap starting.
I20260620 08:18:29.213930 14615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:29.214258 14615 log.cc:824] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:29.214994 14615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca: No bootstrap required, opened a new log
I20260620 08:18:29.215354 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:29.213176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:38573 (local address 127.0.0.1:34009)
0620 08:18:29.213357 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:29.213366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:29.213397 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:18:29.214069 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:29.214080 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:29.214118 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:29.214275 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:29.214950 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:29.214954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:29.214954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:29.214958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:29.215009 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:29.215019 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:29.215112 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:29.215150 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 08:18:29.215191 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260620 08:18:29.216202 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:29.217269 14615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5387132bc2304ea6bfa887ae12a5ddca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260620 08:18:29.217435 14615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:29.217480 14615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5387132bc2304ea6bfa887ae12a5ddca, State: Initialized, Role: FOLLOWER
I20260620 08:18:29.217612 14615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5387132bc2304ea6bfa887ae12a5ddca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260620 08:18:29.217725 14615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:29.217767 14615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:29.217823 14615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:29.218564 14615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5387132bc2304ea6bfa887ae12a5ddca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260620 08:18:29.218680 14615 leader_election.cc:302] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5387132bc2304ea6bfa887ae12a5ddca; no voters:
I20260620 08:18:29.218791 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:29.217510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.31:58387 (local address 127.0.0.1:34009)
0620 08:18:29.217549 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:29.217550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:29.217785 (+ 235us) server_negotiation.cc:407] Connection header received
0620 08:18:29.217826 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:29.217827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:29.217832 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:29.217932 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:29.218518 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:29.218519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:29.218519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:29.218523 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:29.218536 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:29.218540 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:29.218586 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:29.218661 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 08:18:29.218675 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:18:29.218911 14615 leader_election.cc:288] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:29.218940 14625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:29.219074 14625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [term 1 LEADER]: Becoming Leader. State: Replica: 5387132bc2304ea6bfa887ae12a5ddca, State: Running, Role: LEADER
I20260620 08:18:29.219137 14615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:29.219161 14625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5387132bc2304ea6bfa887ae12a5ddca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260620 08:18:29.219462 14626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5387132bc2304ea6bfa887ae12a5ddca. Latest consensus state: current_term: 1 leader_uuid: "5387132bc2304ea6bfa887ae12a5ddca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5387132bc2304ea6bfa887ae12a5ddca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260620 08:18:29.219468 14627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5387132bc2304ea6bfa887ae12a5ddca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5387132bc2304ea6bfa887ae12a5ddca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260620 08:18:29.219642 14627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:29.219866 14626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:29.219875 14632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:29.220578 14632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:29.222317 14632 catalog_manager.cc:1347] Generated new cluster ID: a5c5b9e9b56d419ba29c85d61eab5ca3
I20260620 08:18:29.222378 14632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:29.239262 14632 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:29.239481 14632 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:29.243729 14632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5387132bc2304ea6bfa887ae12a5ddca: Generated new TSK 0
I20260620 08:18:29.243903 14632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:29.249533 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14547
2026-06-20T08:18:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:29 dist-test-slave-65w8 krb5kdc[14646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:29 dist-test-slave-65w8 krb5kdc[14646](info): set up 2 sockets
Jun 20 08:18:29 dist-test-slave-65w8 krb5kdc[14646](info): commencing operation
krb5kdc: starting...
W20260620 08:18:30.326193 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:30 dist-test-slave-65w8 krb5kdc[14646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:30.457587 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43781
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:30.546290 14659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:30.546511 14659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:30.546535 14659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:30.547655 14659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:30.547696 14659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:30.547711 14659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:30.547724 14659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:30.547736 14659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:30.549261 14659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:43781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:30.549484 14659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:30.549667 14659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:30.554883 14665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:30.554926 14659 server_base.cc:1034] running on GCE node
W20260620 08:18:30.554926 14664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:30.555022 14667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:30.555372 14659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:30.555739 14659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:30.556921 14659 hybrid_clock.cc:648] HybridClock initialized: now 1781943510556893 us; error 36 us; skew 500 ppm
Jun 20 08:18:30 dist-test-slave-65w8 krb5kdc[14646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943510, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:30.559505 14659 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:30.559953 14659 webserver.cc:460] Webserver started at http://127.24.249.190:42089/ using document root <none> and password file <none>
I20260620 08:18:30.560202 14659 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:30.560273 14659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:30.560426 14659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:30.561539 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "9a48885b30d94398bfb4a8c922b0dbad"
format_stamp: "Formatted at 2026-06-20 08:18:30 on dist-test-slave-65w8"
server_key: "260b59527c7dac505dde74c68a614b63"
server_key_iv: "8708d4e30cfca0970d9520860e67e1bc"
server_key_version: "encryptionkey@0"
I20260620 08:18:30.561978 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "9a48885b30d94398bfb4a8c922b0dbad"
format_stamp: "Formatted at 2026-06-20 08:18:30 on dist-test-slave-65w8"
server_key: "260b59527c7dac505dde74c68a614b63"
server_key_iv: "8708d4e30cfca0970d9520860e67e1bc"
server_key_version: "encryptionkey@0"
I20260620 08:18:30.563418 14659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:30.564239 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:30.564443 14659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:30.564553 14659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "9a48885b30d94398bfb4a8c922b0dbad"
format_stamp: "Formatted at 2026-06-20 08:18:30 on dist-test-slave-65w8"
server_key: "260b59527c7dac505dde74c68a614b63"
server_key_iv: "8708d4e30cfca0970d9520860e67e1bc"
server_key_version: "encryptionkey@0"
I20260620 08:18:30.564630 14659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:30.571871 14659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:30.573822 14659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:30.574009 14659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:30.577791 14659 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:43781
I20260620 08:18:30.577910 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:43781 every 8 connection(s)
I20260620 08:18:30.578265 14659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:30.579175 14727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:30.581451 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad: Bootstrap starting.
I20260620 08:18:30.582074 14727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:30.582314 14727 log.cc:824] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:30.582988 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad: No bootstrap required, opened a new log
I20260620 08:18:30.583310 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14659
I20260620 08:18:30.583458 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:30.583626 25574 external_mini_cluster.cc:1428] Setting key 0c2173785657867a77f45eeca04b6149
I20260620 08:18:30.585791 14727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a48885b30d94398bfb4a8c922b0dbad" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43781 } }
I20260620 08:18:30.585990 14727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:30.586030 14727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a48885b30d94398bfb4a8c922b0dbad, State: Initialized, Role: FOLLOWER
I20260620 08:18:30.586184 14727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a48885b30d94398bfb4a8c922b0dbad" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43781 } }
I20260620 08:18:30.586251 14727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:30.586272 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:30.586328 14727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:30.586915 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a48885b30d94398bfb4a8c922b0dbad" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43781 } }
I20260620 08:18:30.587026 14727 leader_election.cc:302] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a48885b30d94398bfb4a8c922b0dbad; no voters:
I20260620 08:18:30.587169 14727 leader_election.cc:288] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:30.587244 14732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:30.587421 14727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:30.587440 14732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [term 1 LEADER]: Becoming Leader. State: Replica: 9a48885b30d94398bfb4a8c922b0dbad, State: Running, Role: LEADER
I20260620 08:18:30.587603 14732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a48885b30d94398bfb4a8c922b0dbad" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43781 } }
I20260620 08:18:30.587988 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a48885b30d94398bfb4a8c922b0dbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a48885b30d94398bfb4a8c922b0dbad" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43781 } } }
I20260620 08:18:30.588068 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:30.588235 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a48885b30d94398bfb4a8c922b0dbad. Latest consensus state: current_term: 1 leader_uuid: "9a48885b30d94398bfb4a8c922b0dbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a48885b30d94398bfb4a8c922b0dbad" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 43781 } } }
I20260620 08:18:30.588347 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad [sys.catalog]: This master's current role is: LEADER
W20260620 08:18:30.589227 14748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:30.589299 14748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:30.589382 14741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:30.589572 14741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:30.591312 14741 catalog_manager.cc:1347] Generated new cluster ID: 53a99c921823477ab6db30f8bacf3df7
I20260620 08:18:30.591362 14741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:18:30 dist-test-slave-65w8 krb5kdc[14646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:30.596692 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:30.584233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53389 (local address 127.24.249.190:43781)
0620 08:18:30.584431 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:30.584440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:30.584473 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:18:30.585742 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:30.585751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:30.585812 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:30.586000 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:30.586863 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:30.588293 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:30.589449 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:30.590103 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:30.590226 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:30.592879 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:30.592884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:30.592884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:30.592895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:30.595088 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:30.595529 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:30.595532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:30.595532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:30.595623 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:30.595867 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:30.595868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:30.595868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:30.596101 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:30.596293 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:30.596402 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 08:18:30.596504 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260620 08:18:30.597684 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:30.604417 14741 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:30.605115 14741 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:30.609054 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:30.599172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56595 (local address 127.24.249.190:43781)
0620 08:18:30.599210 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:30.599212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:30.602095 (+ 2883us) server_negotiation.cc:407] Connection header received
0620 08:18:30.602178 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:30.602179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:30.602189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:30.602287 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:30.602859 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:30.603392 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:30.604245 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:30.604535 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:30.604598 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:30.606184 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:30.606185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:30.606186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:30.606193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:30.607782 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:30.608252 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:30.608253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:30.608253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:30.608308 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:30.608548 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:30.608549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:30.608549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:30.608661 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:30.608751 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:30.608907 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 08:18:30.608967 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 08:18:30.611928 14741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a48885b30d94398bfb4a8c922b0dbad: Generated new TSK 0
I20260620 08:18:30.612116 14741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:30.620654 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14659
W20260620 08:18:30.625599 14644 connection.cc:535] client connection to 127.24.249.190:43781 recv error: Network error: failed to read from TLS socket (remote: 127.24.249.190:43781): Connection reset by peer (error 104)
2026-06-20T08:18:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:30 dist-test-slave-65w8 krb5kdc[14758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:30 dist-test-slave-65w8 krb5kdc[14758](info): set up 2 sockets
Jun 20 08:18:30 dist-test-slave-65w8 krb5kdc[14758](info): commencing operation
krb5kdc: starting...
W20260620 08:18:31.682267 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:31 dist-test-slave-65w8 krb5kdc[14758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:31.808346 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41931
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:31.892982 14771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:31.893224 14771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:31.893260 14771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:31.894543 14771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:31.894621 14771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:31.894645 14771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:31.894672 14771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:31.894704 14771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:31.896255 14771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:41931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:31.896556 14771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:31.896799 14771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:31.901770 14779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:31.901754 14776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:31.901750 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:31.902133 14771 server_base.cc:1034] running on GCE node
I20260620 08:18:31.902383 14771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:31.902716 14771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:31.903893 14771 hybrid_clock.cc:648] HybridClock initialized: now 1781943511903860 us; error 52 us; skew 500 ppm
Jun 20 08:18:31 dist-test-slave-65w8 krb5kdc[14758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943511, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:31.911496 14771 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:31.912015 14771 webserver.cc:460] Webserver started at http://127.24.249.190:35211/ using document root <none> and password file <none>
I20260620 08:18:31.912282 14771 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:31.912354 14771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:31.912508 14771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:31.913962 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "cffd8a75d435479f8c2c3139b1cc6c6d"
format_stamp: "Formatted at 2026-06-20 08:18:31 on dist-test-slave-65w8"
server_key: "cce5d27add32ba7365385f94f7a9d0a5"
server_key_iv: "c04833a7afd8613cd1784843ad8109ef"
server_key_version: "encryptionkey@0"
I20260620 08:18:31.914405 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "cffd8a75d435479f8c2c3139b1cc6c6d"
format_stamp: "Formatted at 2026-06-20 08:18:31 on dist-test-slave-65w8"
server_key: "cce5d27add32ba7365385f94f7a9d0a5"
server_key_iv: "c04833a7afd8613cd1784843ad8109ef"
server_key_version: "encryptionkey@0"
I20260620 08:18:31.915807 14771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:31.916672 14786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:31.917063 14771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:31.917155 14771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "cffd8a75d435479f8c2c3139b1cc6c6d"
format_stamp: "Formatted at 2026-06-20 08:18:31 on dist-test-slave-65w8"
server_key: "cce5d27add32ba7365385f94f7a9d0a5"
server_key_iv: "c04833a7afd8613cd1784843ad8109ef"
server_key_version: "encryptionkey@0"
I20260620 08:18:31.917271 14771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:31.935729 14771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:31.938192 14771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:31.938421 14771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:31.942751 14771 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:41931
I20260620 08:18:31.942785 14838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:41931 every 8 connection(s)
I20260620 08:18:31.943130 14771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:31.943802 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14771
I20260620 08:18:31.943948 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:31.944142 25574 external_mini_cluster.cc:1428] Setting key e6cff850f71890594f1275bedd83fa8f
I20260620 08:18:31.944196 14839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:31.947749 14839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d: Bootstrap starting.
I20260620 08:18:31.948385 14839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:31.948681 14839 log.cc:824] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:31.949467 14839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d: No bootstrap required, opened a new log
I20260620 08:18:31.950893 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:31.944711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34393 (local address 127.24.249.190:41931)
0620 08:18:31.945013 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:31.945023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:31.945060 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:18:31.945818 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:31.945833 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:31.945928 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:31.946074 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:31.946291 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:31.947632 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:31.948505 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:31.949079 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:31.949219 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:31.949954 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:31.949958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:31.949958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:31.950161 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:31.950202 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:31.950216 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:31.950303 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:31.950644 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 08:18:31.950674 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":159,"threads_started":1}
I20260620 08:18:31.952042 14839 raft_consensus.cc:348] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41931 } }
I20260620 08:18:31.952232 14839 raft_consensus.cc:374] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:31.952280 14839 raft_consensus.cc:729] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffd8a75d435479f8c2c3139b1cc6c6d, State: Initialized, Role: FOLLOWER
I20260620 08:18:31.952278 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:31.952409 14839 consensus_queue.cc:260] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41931 } }
I20260620 08:18:31.952507 14839 raft_consensus.cc:388] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:31.952549 14839 raft_consensus.cc:482] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:31.952597 14839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:31.953133 14839 raft_consensus.cc:504] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41931 } }
I20260620 08:18:31.953248 14839 leader_election.cc:302] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cffd8a75d435479f8c2c3139b1cc6c6d; no voters:
I20260620 08:18:31.953426 14839 leader_election.cc:288] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:31.953452 14847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:31.953554 14847 raft_consensus.cc:686] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [term 1 LEADER]: Becoming Leader. State: Replica: cffd8a75d435479f8c2c3139b1cc6c6d, State: Running, Role: LEADER
I20260620 08:18:31.953681 14839 sys_catalog.cc:564] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:31.953660 14847 consensus_queue.cc:237] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41931 } }
I20260620 08:18:31.954066 14849 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cffd8a75d435479f8c2c3139b1cc6c6d. Latest consensus state: current_term: 1 leader_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41931 } } }
I20260620 08:18:31.954149 14849 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:31.954308 14850 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffd8a75d435479f8c2c3139b1cc6c6d" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41931 } } }
I20260620 08:18:31.954361 14850 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:31.954378 14853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:31.954849 14853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:31.957296 14853 catalog_manager.cc:1347] Generated new cluster ID: 30bff22136614abbaf19f68077ee2ae5
I20260620 08:18:31.957379 14853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:31.957468 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:31.953718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46133 (local address 127.24.249.190:41931)
0620 08:18:31.953757 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:31.953759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:31.953904 (+ 145us) server_negotiation.cc:407] Connection header received
0620 08:18:31.954009 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:31.954010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:31.954020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:31.954050 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:31.954252 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:31.954987 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:31.955934 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:31.956204 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:31.956277 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:31.957105 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:31.957107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:31.957107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:31.957112 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:31.957128 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:31.957132 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:31.957197 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:31.957382 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 08:18:31.957395 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:31.960708 14853 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:31.961189 14853 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:31.967758 14853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cffd8a75d435479f8c2c3139b1cc6c6d: Generated new TSK 0
I20260620 08:18:31.967973 14853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:31.972965 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14771
W20260620 08:18:31.977301 14756 connection.cc:535] client connection to 127.24.249.190:41931 recv error: Network error: failed to read from TLS socket (remote: 127.24.249.190:41931): Connection reset by peer (error 104)
2026-06-20T08:18:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:32 dist-test-slave-65w8 krb5kdc[14870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:32 dist-test-slave-65w8 krb5kdc[14870](info): set up 2 sockets
Jun 20 08:18:32 dist-test-slave-65w8 krb5kdc[14870](info): commencing operation
krb5kdc: starting...
W20260620 08:18:33.018204 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:33 dist-test-slave-65w8 krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:33.148734 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40709
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:40709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:33.231418 14883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:33.231619 14883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:33.231642 14883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:33.232797 14883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:33.232842 14883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:33.232857 14883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:33.232870 14883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:33.232883 14883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:33.234484 14883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:40709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:33.234730 14883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:33.234969 14883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:33.239913 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:33.239928 14889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:33.239913 14888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:33.240182 14883 server_base.cc:1034] running on GCE node
I20260620 08:18:33.240475 14883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:33.240726 14883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:33.241910 14883 hybrid_clock.cc:648] HybridClock initialized: now 1781943513241895 us; error 30 us; skew 500 ppm
Jun 20 08:18:33 dist-test-slave-65w8 krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943513, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:33.244408 14883 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:33.244789 14883 webserver.cc:460] Webserver started at http://127.24.249.190:39589/ using document root <none> and password file <none>
I20260620 08:18:33.244984 14883 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:33.245023 14883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:33.245108 14883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:33.246189 14883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "c4153821cabf4057a096a412aed25d01"
format_stamp: "Formatted at 2026-06-20 08:18:33 on dist-test-slave-65w8"
server_key: "9ee9291eb73196ad2ae16732b25b5ff4"
server_key_iv: "d8abdc4641b9ed98bbcb55769210adb3"
server_key_version: "encryptionkey@0"
I20260620 08:18:33.246495 14883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "c4153821cabf4057a096a412aed25d01"
format_stamp: "Formatted at 2026-06-20 08:18:33 on dist-test-slave-65w8"
server_key: "9ee9291eb73196ad2ae16732b25b5ff4"
server_key_iv: "d8abdc4641b9ed98bbcb55769210adb3"
server_key_version: "encryptionkey@0"
I20260620 08:18:33.247717 14883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:33.248435 14898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:33.248616 14883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:33.248725 14883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "c4153821cabf4057a096a412aed25d01"
format_stamp: "Formatted at 2026-06-20 08:18:33 on dist-test-slave-65w8"
server_key: "9ee9291eb73196ad2ae16732b25b5ff4"
server_key_iv: "d8abdc4641b9ed98bbcb55769210adb3"
server_key_version: "encryptionkey@0"
I20260620 08:18:33.248802 14883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:33.252265 14883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:33.254267 14883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:33.254433 14883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:33.258523 14883 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:40709
I20260620 08:18:33.258554 14950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:40709 every 8 connection(s)
I20260620 08:18:33.258879 14883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:33.259971 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:33.262318 14951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01: Bootstrap starting.
I20260620 08:18:33.262893 14951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:33.263190 14951 log.cc:824] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:33.263720 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14883
I20260620 08:18:33.263841 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:33.263983 14951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01: No bootstrap required, opened a new log
I20260620 08:18:33.264024 25574 external_mini_cluster.cc:1428] Setting key b4c303349d1bbc8700cb4d18987175de
I20260620 08:18:33.265758 14951 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4153821cabf4057a096a412aed25d01" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40709 } }
I20260620 08:18:33.265981 14951 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:33.266049 14951 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4153821cabf4057a096a412aed25d01, State: Initialized, Role: FOLLOWER
I20260620 08:18:33.266175 14951 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4153821cabf4057a096a412aed25d01" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40709 } }
I20260620 08:18:33.266285 14951 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:33.266330 14951 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:33.266381 14951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:33.267225 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:33.264571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54927 (local address 127.24.249.190:40709)
0620 08:18:33.264790 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:33.264799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:33.264834 (+ 35us) server_negotiation.cc:407] Connection header received
0620 08:18:33.265751 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:33.265758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:33.265789 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:33.265975 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:33.266798 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:33.266801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:33.266802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:33.266806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:33.266858 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:33.266869 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:33.266940 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:33.267015 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 08:18:33.267058 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260620 08:18:33.267377 14951 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4153821cabf4057a096a412aed25d01" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40709 } }
I20260620 08:18:33.267482 14951 leader_election.cc:302] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4153821cabf4057a096a412aed25d01; no voters:
I20260620 08:18:33.267683 14951 leader_election.cc:288] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:33.267706 14956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:33.267851 14956 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [term 1 LEADER]: Becoming Leader. State: Replica: c4153821cabf4057a096a412aed25d01, State: Running, Role: LEADER
I20260620 08:18:33.267982 14951 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:33.268083 14956 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4153821cabf4057a096a412aed25d01" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40709 } }
I20260620 08:18:33.268234 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:33.270121 14957 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4153821cabf4057a096a412aed25d01. Latest consensus state: current_term: 1 leader_uuid: "c4153821cabf4057a096a412aed25d01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4153821cabf4057a096a412aed25d01" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40709 } } }
I20260620 08:18:33.270226 14957 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:33.270576 14975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:33.270665 14958 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4153821cabf4057a096a412aed25d01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4153821cabf4057a096a412aed25d01" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 40709 } } }
I20260620 08:18:33.270723 14958 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:33.271155 14975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:33.271616 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:33.270576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57683 (local address 127.24.249.190:40709)
0620 08:18:33.270613 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:33.270616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:33.270626 (+ 10us) server_negotiation.cc:407] Connection header received
0620 08:18:33.270649 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:33.270650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:33.270655 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:33.270713 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:33.271383 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:33.271384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:33.271384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:33.271389 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:33.271403 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:33.271407 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:33.271454 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:33.271540 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 08:18:33.271555 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:18:33.272888 14975 catalog_manager.cc:1347] Generated new cluster ID: 437f1b1fd26f4bca9a1fe76b20a79610
I20260620 08:18:33.272938 14975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:33.283800 14975 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:33.283998 14975 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:33.291428 14975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4153821cabf4057a096a412aed25d01: Generated new TSK 0
I20260620 08:18:33.291632 14975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:33.298111 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14883
2026-06-20T08:18:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:33 dist-test-slave-65w8 krb5kdc[14983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:33 dist-test-slave-65w8 krb5kdc[14983](info): set up 2 sockets
Jun 20 08:18:33 dist-test-slave-65w8 krb5kdc[14983](info): commencing operation
krb5kdc: starting...
W20260620 08:18:34.362377 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:34 dist-test-slave-65w8 krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:34.488895 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:38003
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:38003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:34.569849 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:34.570147 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:34.570188 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:34.571368 14996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:34.571441 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:34.571466 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:34.571492 14996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:34.571524 14996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:34.573012 14996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:38003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:38003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:34.573302 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:34.573534 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:34.578300 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:34.578299 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:34.578441 14996 server_base.cc:1034] running on GCE node
W20260620 08:18:34.578369 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:34.578718 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:34.579044 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:34.580240 14996 hybrid_clock.cc:648] HybridClock initialized: now 1781943514580242 us; error 49 us; skew 500 ppm
Jun 20 08:18:34 dist-test-slave-65w8 krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943514, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:34.583036 14996 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:34.583596 14996 webserver.cc:460] Webserver started at http://127.24.249.190:33601/ using document root <none> and password file <none>
I20260620 08:18:34.583848 14996 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:34.583923 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:34.584080 14996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:34.585263 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "e9d8a105b3bf49f38610679ee276225c"
format_stamp: "Formatted at 2026-06-20 08:18:34 on dist-test-slave-65w8"
server_key: "54701434dba3a76fc35ad8a997f92432"
server_key_iv: "a4c7fcf660d6e298fea27b4c666234bb"
server_key_version: "encryptionkey@0"
I20260620 08:18:34.585670 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "e9d8a105b3bf49f38610679ee276225c"
format_stamp: "Formatted at 2026-06-20 08:18:34 on dist-test-slave-65w8"
server_key: "54701434dba3a76fc35ad8a997f92432"
server_key_iv: "a4c7fcf660d6e298fea27b4c666234bb"
server_key_version: "encryptionkey@0"
I20260620 08:18:34.587306 14996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:34.588284 15011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:34.588600 14996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:34.588747 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "e9d8a105b3bf49f38610679ee276225c"
format_stamp: "Formatted at 2026-06-20 08:18:34 on dist-test-slave-65w8"
server_key: "54701434dba3a76fc35ad8a997f92432"
server_key_iv: "a4c7fcf660d6e298fea27b4c666234bb"
server_key_version: "encryptionkey@0"
I20260620 08:18:34.588819 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:34.602425 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:34.604818 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:34.604981 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:34.609025 14996 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:38003
I20260620 08:18:34.609061 15063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:38003 every 8 connection(s)
I20260620 08:18:34.609443 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:34.610400 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:34.612604 15064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c: Bootstrap starting.
I20260620 08:18:34.613178 15064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:34.613456 15064 log.cc:824] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:34.614086 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 14996
I20260620 08:18:34.614219 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:34.614203 15064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c: No bootstrap required, opened a new log
I20260620 08:18:34.614391 25574 external_mini_cluster.cc:1428] Setting key 7e5a3e1ef1898d45e970f283bdd30e18
I20260620 08:18:34.616523 15064 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8a105b3bf49f38610679ee276225c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38003 } }
I20260620 08:18:34.616699 15064 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:34.616739 15064 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d8a105b3bf49f38610679ee276225c, State: Initialized, Role: FOLLOWER
I20260620 08:18:34.616926 15064 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8a105b3bf49f38610679ee276225c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38003 } }
I20260620 08:18:34.617007 15064 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:34.617036 15064 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:34.617079 15064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:34.617743 15064 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8a105b3bf49f38610679ee276225c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38003 } }
I20260620 08:18:34.617846 15064 leader_election.cc:302] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d8a105b3bf49f38610679ee276225c; no voters:
I20260620 08:18:34.618016 15064 leader_election.cc:288] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:34.618073 15069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:34.618211 15069 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [term 1 LEADER]: Becoming Leader. State: Replica: e9d8a105b3bf49f38610679ee276225c, State: Running, Role: LEADER
I20260620 08:18:34.618263 15064 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:34.618372 15069 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8a105b3bf49f38610679ee276225c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38003 } }
I20260620 08:18:34.618685 15069 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d8a105b3bf49f38610679ee276225c. Latest consensus state: current_term: 1 leader_uuid: "e9d8a105b3bf49f38610679ee276225c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8a105b3bf49f38610679ee276225c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38003 } } }
I20260620 08:18:34.618818 15069 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:34.619059 15076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:34.619541 15076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:34.620455 15070 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d8a105b3bf49f38610679ee276225c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8a105b3bf49f38610679ee276225c" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 38003 } } }
I20260620 08:18:34.620548 15070 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:34.621508 15076 catalog_manager.cc:1347] Generated new cluster ID: 16d5e080c7bd4c9dbf7356ca5faa299c
I20260620 08:18:34.621557 15076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 08:18:34 dist-test-slave-65w8 krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:34.626582 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:34.614923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56377 (local address 127.24.249.190:38003)
0620 08:18:34.615100 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:34.615110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:34.615142 (+ 32us) server_negotiation.cc:407] Connection header received
0620 08:18:34.615710 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:34.615719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:34.615779 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:34.615937 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:34.616547 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:34.618223 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:34.619372 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:34.619943 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:34.620055 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:34.622893 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:34.622897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:34.622897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:34.622909 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:34.624751 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:34.625175 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:34.625178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:34.625178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:34.625278 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:34.625648 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:34.625649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:34.625649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:34.625947 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:34.626070 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:34.626236 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 08:18:34.626370 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260620 08:18:34.627442 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:34.636011 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:34.628801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55723 (local address 127.24.249.190:38003)
0620 08:18:34.628834 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:34.628836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:34.629066 (+ 230us) server_negotiation.cc:407] Connection header received
0620 08:18:34.629145 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:34.629146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:34.629154 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:34.629179 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:34.629796 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:34.630557 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:34.631527 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:34.631839 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:34.631873 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:34.633258 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:34.633259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:34.633259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:34.633265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:34.634819 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:34.635179 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:34.635179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:34.635180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:34.635230 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:34.635432 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:34.635432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:34.635433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:34.635538 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:34.635608 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:34.635830 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 08:18:34.635902 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 08:18:34.636233 15076 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:34.636768 15076 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:34.641475 15076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d8a105b3bf49f38610679ee276225c: Generated new TSK 0
I20260620 08:18:34.641660 15076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:34.643525 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 14996
2026-06-20T08:18:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:34 dist-test-slave-65w8 krb5kdc[15095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:34 dist-test-slave-65w8 krb5kdc[15095](info): set up 2 sockets
Jun 20 08:18:34 dist-test-slave-65w8 krb5kdc[15095](info): commencing operation
krb5kdc: starting...
W20260620 08:18:35.734555 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:35 dist-test-slave-65w8 krb5kdc[15095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:35.862295 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41899
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:41899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:35.946843 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:35.947068 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:35.947093 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:35.948355 15108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:35.948406 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:35.948421 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:35.948433 15108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:35.948446 15108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:35.950071 15108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:41899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:41899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:35.950341 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:35.950595 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:35.956569 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:35.956543 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:35.956915 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:35.957759 15108 server_base.cc:1034] running on GCE node
I20260620 08:18:35.958089 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:35.958532 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:35.959693 15108 hybrid_clock.cc:648] HybridClock initialized: now 1781943515959671 us; error 41 us; skew 500 ppm
Jun 20 08:18:35 dist-test-slave-65w8 krb5kdc[15095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943515, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:35.963464 15108 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:35.963980 15108 webserver.cc:460] Webserver started at http://127.24.249.190:35573/ using document root <none> and password file <none>
I20260620 08:18:35.964314 15108 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:35.964409 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:35.964560 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:35.965766 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "dd502253f0c9480c9367524b9ef4029f"
format_stamp: "Formatted at 2026-06-20 08:18:35 on dist-test-slave-65w8"
server_key: "86b87fc0697f0fe60833f8b91b62d3c8"
server_key_iv: "a4687c1c0ac9fc8ab73bc2bfa020e7d0"
server_key_version: "encryptionkey@0"
I20260620 08:18:35.966248 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "dd502253f0c9480c9367524b9ef4029f"
format_stamp: "Formatted at 2026-06-20 08:18:35 on dist-test-slave-65w8"
server_key: "86b87fc0697f0fe60833f8b91b62d3c8"
server_key_iv: "a4687c1c0ac9fc8ab73bc2bfa020e7d0"
server_key_version: "encryptionkey@0"
I20260620 08:18:35.967769 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 08:18:35.968710 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:35.969002 15108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:35.969125 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "dd502253f0c9480c9367524b9ef4029f"
format_stamp: "Formatted at 2026-06-20 08:18:35 on dist-test-slave-65w8"
server_key: "86b87fc0697f0fe60833f8b91b62d3c8"
server_key_iv: "a4687c1c0ac9fc8ab73bc2bfa020e7d0"
server_key_version: "encryptionkey@0"
I20260620 08:18:35.969260 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:35.988088 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:35.991276 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:35.991478 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:35.998003 15108 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:41899
I20260620 08:18:35.998145 15175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:41899 every 8 connection(s)
I20260620 08:18:35.998682 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:35.999823 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:36.002712 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f: Bootstrap starting.
I20260620 08:18:36.003585 15176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:36.003954 15176 log.cc:824] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:36.004868 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f: No bootstrap required, opened a new log
I20260620 08:18:36.006808 15176 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd502253f0c9480c9367524b9ef4029f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41899 } }
I20260620 08:18:36.006969 15176 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:36.006994 15176 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd502253f0c9480c9367524b9ef4029f, State: Initialized, Role: FOLLOWER
I20260620 08:18:36.007155 15176 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd502253f0c9480c9367524b9ef4029f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41899 } }
I20260620 08:18:36.007242 15176 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:36.007267 15176 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:36.007306 15176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:36.008273 15176 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd502253f0c9480c9367524b9ef4029f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41899 } }
I20260620 08:18:36.008426 15176 leader_election.cc:302] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd502253f0c9480c9367524b9ef4029f; no voters:
I20260620 08:18:36.008461 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15108
I20260620 08:18:36.008579 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:36.008641 15176 leader_election.cc:288] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:36.008703 15179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:36.008797 25574 external_mini_cluster.cc:1428] Setting key ac9255ea435525cc2219d2933148f9e2
I20260620 08:18:36.008867 15179 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [term 1 LEADER]: Becoming Leader. State: Replica: dd502253f0c9480c9367524b9ef4029f, State: Running, Role: LEADER
I20260620 08:18:36.008939 15176 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:36.008975 15179 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd502253f0c9480c9367524b9ef4029f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41899 } }
I20260620 08:18:36.009426 15180 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd502253f0c9480c9367524b9ef4029f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd502253f0c9480c9367524b9ef4029f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41899 } } }
I20260620 08:18:36.009512 15180 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:36.009686 15181 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd502253f0c9480c9367524b9ef4029f. Latest consensus state: current_term: 1 leader_uuid: "dd502253f0c9480c9367524b9ef4029f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd502253f0c9480c9367524b9ef4029f" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 41899 } } }
I20260620 08:18:36.009765 15181 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:36.009982 15188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:36.010664 15188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:36.013211 15188 catalog_manager.cc:1347] Generated new cluster ID: ac66c61876f64d359762d06e08f661f5
I20260620 08:18:36.013280 15188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:36.016505 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:36.009514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50755 (local address 127.24.249.190:41899)
0620 08:18:36.009978 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:36.009987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:36.010020 (+ 33us) server_negotiation.cc:407] Connection header received
0620 08:18:36.010154 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:36.010162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:36.010226 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:36.010391 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:36.010643 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:36.012520 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:36.014042 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:36.014511 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:36.014882 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:36.015739 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:36.015744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:36.015744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:36.015985 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:36.016021 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:36.016035 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:36.016128 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:36.016216 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 08:18:36.016295 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":67,"threads_started":1}
I20260620 08:18:36.017696 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:36.023422 15188 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:36.024259 15188 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:36.027112 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:36.019317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37537 (local address 127.24.249.190:41899)
0620 08:18:36.019360 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:36.019362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:36.022882 (+ 3520us) server_negotiation.cc:407] Connection header received
0620 08:18:36.023037 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:36.023039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:36.023055 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:36.023151 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:36.023375 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:36.024218 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:36.025527 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:36.025903 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:36.025967 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:36.026808 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:36.026809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:36.026810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:36.026816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:36.026834 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:36.026840 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:36.026898 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:36.026972 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 08:18:36.027024 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:18:36.033257 15188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd502253f0c9480c9367524b9ef4029f: Generated new TSK 0
I20260620 08:18:36.033494 15188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:36.036753 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15108
2026-06-20T08:18:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:36 dist-test-slave-65w8 krb5kdc[15207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:36 dist-test-slave-65w8 krb5kdc[15207](info): set up 2 sockets
Jun 20 08:18:36 dist-test-slave-65w8 krb5kdc[15207](info): commencing operation
krb5kdc: starting...
W20260620 08:18:37.082705 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:37 dist-test-slave-65w8 krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:37.213968 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:44995
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:37.297237 15220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:37.297508 15220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:37.297547 15220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:37.298826 15220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:37.298900 15220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:37.298926 15220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:37.298954 15220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:37.299008 15220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:37.300529 15220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:44995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:37.300807 15220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:37.301047 15220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:37.306241 15226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:37.306286 15228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:37.306348 15220 server_base.cc:1034] running on GCE node
W20260620 08:18:37.306530 15225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:37.306947 15220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:37.307267 15220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:37.308449 15220 hybrid_clock.cc:648] HybridClock initialized: now 1781943517308436 us; error 31 us; skew 500 ppm
Jun 20 08:18:37 dist-test-slave-65w8 krb5kdc[15207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943517, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:37.315478 15220 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:37.315981 15220 webserver.cc:460] Webserver started at http://127.24.249.190:40731/ using document root <none> and password file <none>
I20260620 08:18:37.316241 15220 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:37.316308 15220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:37.316434 15220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:37.317567 15220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "0ce1d848086f4673a84a5f69acc683bc"
format_stamp: "Formatted at 2026-06-20 08:18:37 on dist-test-slave-65w8"
server_key: "aa6fff5325cae56a4e6e8a39ffb5359a"
server_key_iv: "7c22d4ceef03543af5803d4a011e6637"
server_key_version: "encryptionkey@0"
I20260620 08:18:37.317988 15220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "0ce1d848086f4673a84a5f69acc683bc"
format_stamp: "Formatted at 2026-06-20 08:18:37 on dist-test-slave-65w8"
server_key: "aa6fff5325cae56a4e6e8a39ffb5359a"
server_key_iv: "7c22d4ceef03543af5803d4a011e6637"
server_key_version: "encryptionkey@0"
I20260620 08:18:37.319473 15220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:37.320425 15235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:37.320703 15220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:37.320832 15220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "0ce1d848086f4673a84a5f69acc683bc"
format_stamp: "Formatted at 2026-06-20 08:18:37 on dist-test-slave-65w8"
server_key: "aa6fff5325cae56a4e6e8a39ffb5359a"
server_key_iv: "7c22d4ceef03543af5803d4a011e6637"
server_key_version: "encryptionkey@0"
I20260620 08:18:37.320920 15220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:37.324820 15220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:37.327193 15220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:37.327353 15220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:37.331419 15220 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:44995
I20260620 08:18:37.331460 15287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:44995 every 8 connection(s)
I20260620 08:18:37.331801 15220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:37.332850 15288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:37.335268 15288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc: Bootstrap starting.
I20260620 08:18:37.335830 15288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:37.336072 15288 log.cc:824] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:37.336959 15288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc: No bootstrap required, opened a new log
I20260620 08:18:37.338721 15288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1d848086f4673a84a5f69acc683bc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44995 } }
I20260620 08:18:37.338867 15288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 08:18:37.338891 15288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce1d848086f4673a84a5f69acc683bc, State: Initialized, Role: FOLLOWER
I20260620 08:18:37.339040 15288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1d848086f4673a84a5f69acc683bc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44995 } }
I20260620 08:18:37.339131 15288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 08:18:37.339177 15288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 08:18:37.339227 15288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:37.339282 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15220
I20260620 08:18:37.339380 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:37.339552 25574 external_mini_cluster.cc:1428] Setting key 8045d5790fe0cf406444a013d59f1fb0
I20260620 08:18:37.340605 15288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1d848086f4673a84a5f69acc683bc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44995 } }
I20260620 08:18:37.340723 15288 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce1d848086f4673a84a5f69acc683bc; no voters:
I20260620 08:18:37.340930 15288 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 08:18:37.341025 15293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:37.341205 15288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:37.341572 15293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [term 1 LEADER]: Becoming Leader. State: Replica: 0ce1d848086f4673a84a5f69acc683bc, State: Running, Role: LEADER
I20260620 08:18:37.341671 15293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1d848086f4673a84a5f69acc683bc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44995 } }
I20260620 08:18:37.342154 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:37.340207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43331 (local address 127.24.249.190:44995)
0620 08:18:37.340429 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:37.340439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:37.340476 (+ 37us) server_negotiation.cc:407] Connection header received
0620 08:18:37.340602 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:37.340609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:37.340640 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:37.340850 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:37.341543 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:37.341547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:37.341548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:37.341553 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:37.341602 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:37.341619 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:37.341703 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:37.341837 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 08:18:37.341916 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260620 08:18:37.342272 15294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce1d848086f4673a84a5f69acc683bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1d848086f4673a84a5f69acc683bc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44995 } } }
I20260620 08:18:37.342370 15294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:37.342687 15295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce1d848086f4673a84a5f69acc683bc. Latest consensus state: current_term: 1 leader_uuid: "0ce1d848086f4673a84a5f69acc683bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce1d848086f4673a84a5f69acc683bc" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44995 } } }
I20260620 08:18:37.342765 15295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:37.342896 15303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:37.343387 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:37.343604 15303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:37.345932 15303 catalog_manager.cc:1347] Generated new cluster ID: 30c0b98dbcfe49f692a9bf1b9cc28621
I20260620 08:18:37.345989 15303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:37.346529 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:37.345121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51525 (local address 127.24.249.190:44995)
0620 08:18:37.345155 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:37.345157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:37.345432 (+ 275us) server_negotiation.cc:407] Connection header received
0620 08:18:37.345518 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:37.345519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:37.345526 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:37.345584 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:37.346270 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:37.346272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:37.346272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 08:18:37.346276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:37.346292 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 08:18:37.346297 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:37.346344 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:37.346456 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 08:18:37.346470 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:18:37.357170 15303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:37.357378 15303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:37.364519 15303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce1d848086f4673a84a5f69acc683bc: Generated new TSK 0
I20260620 08:18:37.364770 15303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 08:18:37.372782 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15220
2026-06-20T08:18:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1335 ms)
[----------] 18 tests from AuthTokenIssuingTest (24369 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:37 dist-test-slave-65w8 krb5kdc[15319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:37 dist-test-slave-65w8 krb5kdc[15319](info): set up 2 sockets
Jun 20 08:18:37 dist-test-slave-65w8 krb5kdc[15319](info): commencing operation
krb5kdc: starting...
W20260620 08:18:38.442288 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:38 dist-test-slave-65w8 krb5kdc[15319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:38.568969 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:44947
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:44947,127.24.249.189:42239,127.24.249.188:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:38.652010 15332 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:18:38.652242 15332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:38.652261 15332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:38.652282 15332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:38.653507 15332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:38.653563 15332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:38.653585 15332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:38.653604 15332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:38.653615 15332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:38.655108 15332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:44947,127.24.249.189:42239,127.24.249.188:38825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:38.655328 15332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:38.655509 15332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:38.661007 15338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:38.661007 15340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:38.661034 15337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:38.661463 15332 server_base.cc:1034] running on GCE node
I20260620 08:18:38.661739 15332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:38.662083 15332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:38.663286 15332 hybrid_clock.cc:648] HybridClock initialized: now 1781943518663272 us; error 33 us; skew 500 ppm
Jun 20 08:18:38 dist-test-slave-65w8 krb5kdc[15319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:38.666347 15332 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:38.666885 15332 webserver.cc:460] Webserver started at http://127.24.249.190:43897/ using document root <none> and password file <none>
I20260620 08:18:38.667136 15332 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:38.667207 15332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:38.667362 15332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:38.668476 15332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "21324ad537c84f288dc40977ce254d8e"
format_stamp: "Formatted at 2026-06-20 08:18:38 on dist-test-slave-65w8"
server_key: "ef88d26b706d1717bbde92b461c8c35f"
server_key_iv: "3b8e3bdad8402b5a6b1bea0934cd3583"
server_key_version: "encryptionkey@0"
I20260620 08:18:38.668855 15332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "21324ad537c84f288dc40977ce254d8e"
format_stamp: "Formatted at 2026-06-20 08:18:38 on dist-test-slave-65w8"
server_key: "ef88d26b706d1717bbde92b461c8c35f"
server_key_iv: "3b8e3bdad8402b5a6b1bea0934cd3583"
server_key_version: "encryptionkey@0"
I20260620 08:18:38.670310 15332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 08:18:38.671214 15347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:38.671428 15332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:38.671540 15332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "21324ad537c84f288dc40977ce254d8e"
format_stamp: "Formatted at 2026-06-20 08:18:38 on dist-test-slave-65w8"
server_key: "ef88d26b706d1717bbde92b461c8c35f"
server_key_iv: "3b8e3bdad8402b5a6b1bea0934cd3583"
server_key_version: "encryptionkey@0"
I20260620 08:18:38.671614 15332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:38.709379 15332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:38.712280 15332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:38.712446 15332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:38.717286 15332 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:44947
I20260620 08:18:38.717347 15399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:44947 every 8 connection(s)
I20260620 08:18:38.717705 15332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:38.718873 15400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:38.721426 15400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:38.724628 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15332
I20260620 08:18:38.724752 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
I20260620 08:18:38.724932 25574 external_mini_cluster.cc:1428] Setting key c5a2f8415a473d3d91f4b89e4be2e975
Jun 20 08:18:38 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:38.732714 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.722339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.24.249.190:44947)
0620 08:18:38.722522 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:38.722532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:38.722662 (+ 130us) server_negotiation.cc:407] Connection header received
0620 08:18:38.723481 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:38.723485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:38.723519 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:38.723628 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:38.724370 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.725335 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.726407 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.726713 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.726842 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.729222 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:38.729231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:38.729231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:38.729240 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:38.731009 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.731509 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.731512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.731512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.731597 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.731869 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.731870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.731870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.732117 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:38.732186 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:38.732435 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 08:18:38.732540 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260620 08:18:38.732832 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.722336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:44947 (local address 127.0.0.1:42992)
0620 08:18:38.722510 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.722548 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:38.723223 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:38.723657 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:38.723659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:38.724047 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:38.724261 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.724266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.725434 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.725435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.726312 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.726314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.726816 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.726817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.726881 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.727457 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:38.727466 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:38.729132 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:38.731105 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.731114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.731116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.731413 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.731677 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.731678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.731678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.731793 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.732189 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:38.732196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:38.732274 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 08:18:38.732394 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 08:18:38.732647 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":86,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.24.249.189@KRBTEST.COM; defaulting to no policy
I20260620 08:18:38.733978 15400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:38.734341 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.734210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60782)
0620 08:18:38.734261 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.734303 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:18:38.734463 15348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
I20260620 08:18:38.734930 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.734839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60784)
0620 08:18:38.734885 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.734909 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:18:38.735035 15400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:42239: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab.
Entry for principal kudu/127.24.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab.
I20260620 08:18:38.752611 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:42239
--webserver_interface=127.24.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab
--principal=kudu/127.24.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:44947,127.24.249.189:42239,127.24.249.188:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:18:38.784292 15400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } attempt: 1
I20260620 08:18:38.784965 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.784668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60788)
0620 08:18:38.784878 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.784932 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260620 08:18:38.785403 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.785254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60790)
0620 08:18:38.785344 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.785372 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 08:18:38.785504 15400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:42239: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
W20260620 08:18:38.837388 15407 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:18:38.837682 15407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:38.837726 15407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:38.837774 15407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:38.839107 15407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:38.839191 15407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:38.839219 15407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:38.839251 15407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:38.839285 15407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:38.841144 15407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:44947,127.24.249.189:42239,127.24.249.188:38825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab
--principal=kudu/127.24.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:38.841497 15407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:38.841756 15407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:38.846845 15412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:38.846894 15415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:38.846845 15413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:38.847084 15407 server_base.cc:1034] running on GCE node
I20260620 08:18:38.847374 15407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:38.847692 15407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:38.848891 15407 hybrid_clock.cc:648] HybridClock initialized: now 1781943518848877 us; error 28 us; skew 500 ppm
Jun 20 08:18:38 dist-test-slave-65w8 krb5kdc[15319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:38.851591 15407 init.cc:377] Logged in from keytab as kudu/127.24.249.189@KRBTEST.COM (short username kudu)
I20260620 08:18:38.852020 15407 webserver.cc:460] Webserver started at http://127.24.249.189:38751/ using document root <none> and password file <none>
I20260620 08:18:38.852252 15407 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:38.852321 15407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:38.852473 15407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:38.853531 15407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data/instance:
uuid: "5bc13c2efe834c0498d10a57752dd0a3"
format_stamp: "Formatted at 2026-06-20 08:18:38 on dist-test-slave-65w8"
server_key: "ae2e41c537ef8fbc84d9d6a44706af82"
server_key_iv: "3f7756a2527b9f0684c0fbd02efdeb07"
server_key_version: "encryptionkey@0"
I20260620 08:18:38.853660 15400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } attempt: 2
I20260620 08:18:38.854027 15407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/wal/instance:
uuid: "5bc13c2efe834c0498d10a57752dd0a3"
format_stamp: "Formatted at 2026-06-20 08:18:38 on dist-test-slave-65w8"
server_key: "ae2e41c537ef8fbc84d9d6a44706af82"
server_key_iv: "3f7756a2527b9f0684c0fbd02efdeb07"
server_key_version: "encryptionkey@0"
I20260620 08:18:38.854113 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.853971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60800)
0620 08:18:38.854036 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.854076 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 08:18:38.854470 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.854377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60804)
0620 08:18:38.854419 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.854446 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 08:18:38.854588 15400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:42239: Network error: Client connection negotiation failed: client connection to 127.24.249.189:42239: connect: Connection refused (error 111)
I20260620 08:18:38.855574 15407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:38.856382 15422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:38.856611 15407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:38.856762 15407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/wal
uuid: "5bc13c2efe834c0498d10a57752dd0a3"
format_stamp: "Formatted at 2026-06-20 08:18:38 on dist-test-slave-65w8"
server_key: "ae2e41c537ef8fbc84d9d6a44706af82"
server_key_iv: "3f7756a2527b9f0684c0fbd02efdeb07"
server_key_version: "encryptionkey@0"
I20260620 08:18:38.856848 15407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:38.865594 15407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:38.867982 15407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:38.868148 15407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:38.872352 15407 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.189:42239
I20260620 08:18:38.872393 15474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.189:42239 every 8 connection(s)
I20260620 08:18:38.872782 15407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
I20260620 08:18:38.873919 15475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:38.875784 15475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:38.877938 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15407
I20260620 08:18:38.878046 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-1/wal/instance
I20260620 08:18:38.878242 25574 external_mini_cluster.cc:1428] Setting key 84046bef1dc5a596aef3fc8e6d2c85a8
Jun 20 08:18:38 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.188@KRBTEST.COM; defaulting to no policy
I20260620 08:18:38.886061 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.876524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.24.249.190:44947)
0620 08:18:38.876586 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:38.876588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:38.876800 (+ 212us) server_negotiation.cc:407] Connection header received
0620 08:18:38.877262 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:38.877263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:38.877275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:38.877339 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:38.878129 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.879040 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.880370 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.880610 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.880710 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.882984 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:38.882986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:38.882986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:38.882994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:38.884493 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.884980 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.884981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.884981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.885050 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.885309 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.885310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.885310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.885447 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:38.885529 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:38.885830 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 08:18:38.885964 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 08:18:38.886276 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.876483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:44947 (local address 127.0.0.1:43002)
0620 08:18:38.876661 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.876703 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:38.877179 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:38.877402 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:38.877405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:38.877649 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:38.878011 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.878020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.879142 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.879143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.880269 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.880272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.880730 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.880731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.880858 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.881432 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:38.881440 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:38.882880 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:38.884609 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.884618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.884620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.884879 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.885122 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.885123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.885123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.885235 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.885520 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:38.885527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:38.885674 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 08:18:38.885795 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 08:18:38.886023 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260620 08:18:38.886869 15475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.188@KRBTEST.COM" created.
Jun 20 08:18:38 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:18:38.895269 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.887157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60812)
0620 08:18:38.887367 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.887377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:38.887422 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:38.887573 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:38.887574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:38.887694 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:38.887745 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.887746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.888738 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.888739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.889465 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.889468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.890203 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.890204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.890263 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.890708 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:38.890714 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:38.892138 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:38.894177 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.894178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.894179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.894356 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.894580 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.894581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.894581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.894637 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.894940 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:38.894940 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:38.894979 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 08:18:38.895040 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 08:18:38.895147 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260620 08:18:38.895301 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.887227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.24.249.189:42239)
0620 08:18:38.887384 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:38.887388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:38.887418 (+ 30us) server_negotiation.cc:407] Connection header received
0620 08:18:38.887475 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:38.887477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:38.887488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:38.887549 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:38.887810 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.888623 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.889551 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.890119 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.890196 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.892244 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:38.892253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:38.892253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:38.892260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:38.894081 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.894454 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.894459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.894459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.894514 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.894709 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.894710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.894711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.894866 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:38.894963 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:38.895078 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 08:18:38.895200 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260620 08:18:38.896202 15475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:38.896562 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.896455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47408)
0620 08:18:38.896493 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.896539 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 08:18:38.896695 15426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
I20260620 08:18:38.897145 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.897032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47410)
0620 08:18:38.897092 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.897118 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 08:18:38.897248 15475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:38825: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab.
Entry for principal kudu/127.24.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab.
I20260620 08:18:38.905464 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:38825
--webserver_interface=127.24.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab
--principal=kudu/127.24.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:44947,127.24.249.189:42239,127.24.249.188:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:18:38.945791 15400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } attempt: 3
I20260620 08:18:38.946468 15475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } attempt: 1
I20260620 08:18:38.946834 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.946709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47426)
0620 08:18:38.946766 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.946806 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 08:18:38.947202 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.947107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47430)
0620 08:18:38.947153 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.947177 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 08:18:38.947379 15475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:38825: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Jun 20 08:18:38 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:18:38.953531 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.946249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60822)
0620 08:18:38.946323 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.946341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:38.946416 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:38.946578 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:38.946579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:38.946774 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:38.946860 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.946862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.947826 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.947827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.948378 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:38.948380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.948871 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.948872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:38.948924 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.949376 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:38.949381 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:38.950596 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:38.952497 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.952498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.952499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.952702 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.952939 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:38.952940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:38.952941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:38.953003 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:38.953292 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:38.953292 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:38.953333 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 08:18:38.953388 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 08:18:38.953466 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 08:18:38.953591 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.946334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.24.249.189:42239)
0620 08:18:38.946424 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:38.946427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:38.946443 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:18:38.946479 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:38.946480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:38.946494 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:38.946563 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:38.946960 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.947738 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.948459 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:38.948796 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:38.948868 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:38.950673 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:38.950674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:38.950675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:38.950682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:38.952404 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.952779 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.952780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.952781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.952841 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:38.953072 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:38.953073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:38.953074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:38.953203 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:38.953270 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:38.953432 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 08:18:38.953510 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 08:18:38.954015 15400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:38.954380 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.954272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47436)
0620 08:18:38.954321 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.954347 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 08:18:38.954699 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.954643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47438)
0620 08:18:38.954666 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.954687 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 08:18:38.954806 15400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:38825: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
I20260620 08:18:38.985970 15400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } attempt: 1
I20260620 08:18:38.986524 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.986350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47452)
0620 08:18:38.986440 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.986486 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 08:18:38.987005 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:38.986883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47466)
0620 08:18:38.986951 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:38.986983 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 08:18:38.987135 15400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:38825: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
W20260620 08:18:38.991313 15482 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:18:38.991544 15482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:38.991596 15482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:38.991639 15482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:38.992930 15482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:38.992992 15482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:38.993018 15482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:38.993062 15482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:38.993089 15482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:38.995095 15482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:44947,127.24.249.189:42239,127.24.249.188:38825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab
--principal=kudu/127.24.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:38.995436 15482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:38.995692 15482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:39.001219 15487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:39.001273 15488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:39.001197 15490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:39.001420 15482 server_base.cc:1034] running on GCE node
I20260620 08:18:39.001716 15482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:39.002080 15482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:39.003264 15482 hybrid_clock.cc:648] HybridClock initialized: now 1781943519003230 us; error 50 us; skew 500 ppm
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943519, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:39.006060 15482 init.cc:377] Logged in from keytab as kudu/127.24.249.188@KRBTEST.COM (short username kudu)
I20260620 08:18:39.006551 15482 webserver.cc:460] Webserver started at http://127.24.249.188:34691/ using document root <none> and password file <none>
I20260620 08:18:39.006794 15482 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:39.006863 15482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:39.007025 15482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:39.008177 15482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data/instance:
uuid: "c2f386c8bfe1474e8971a821772e7aa0"
format_stamp: "Formatted at 2026-06-20 08:18:39 on dist-test-slave-65w8"
server_key: "0eb16eedc32de5a9f81997f2c9c229d9"
server_key_iv: "130a12d37dab8423002cd8b8604530ca"
server_key_version: "encryptionkey@0"
I20260620 08:18:39.008550 15482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/wal/instance:
uuid: "c2f386c8bfe1474e8971a821772e7aa0"
format_stamp: "Formatted at 2026-06-20 08:18:39 on dist-test-slave-65w8"
server_key: "0eb16eedc32de5a9f81997f2c9c229d9"
server_key_iv: "130a12d37dab8423002cd8b8604530ca"
server_key_version: "encryptionkey@0"
I20260620 08:18:39.009981 15482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:39.010775 15497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:39.010952 15482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 08:18:39.011053 15482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/wal
uuid: "c2f386c8bfe1474e8971a821772e7aa0"
format_stamp: "Formatted at 2026-06-20 08:18:39 on dist-test-slave-65w8"
server_key: "0eb16eedc32de5a9f81997f2c9c229d9"
server_key_iv: "130a12d37dab8423002cd8b8604530ca"
server_key_version: "encryptionkey@0"
I20260620 08:18:39.011143 15482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:39.015548 15475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } attempt: 2
I20260620 08:18:39.016003 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.015865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47470)
0620 08:18:39.015929 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:39.015972 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 08:18:39.016368 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.016259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47486)
0620 08:18:39.016308 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:39.016337 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 08:18:39.016461 15475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:38825: Network error: Client connection negotiation failed: client connection to 127.24.249.188:38825: connect: Connection refused (error 111)
I20260620 08:18:39.019734 15482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:39.022063 15482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:39.022253 15482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:39.026707 15482 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.188:38825
I20260620 08:18:39.026780 15549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.188:38825 every 8 connection(s)
I20260620 08:18:39.027143 15482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
I20260620 08:18:39.028034 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:39.029569 15550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:39.030853 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15482
I20260620 08:18:39.030970 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781943243671421-25574-0/minicluster-data/master-2/wal/instance
I20260620 08:18:39.031214 25574 external_mini_cluster.cc:1428] Setting key 249b44c7e907cf83d233bdd8e3e803f3
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943519, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:18:39.040781 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.030335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.24.249.190:44947)
0620 08:18:39.030426 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.030429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.030687 (+ 258us) server_negotiation.cc:407] Connection header received
0620 08:18:39.031192 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.031193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.031209 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.031295 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.032238 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.032972 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.034229 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.034515 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.034594 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.037154 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.037156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.037156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.037166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.039039 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.039717 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.039718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.039719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.039790 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.040057 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.040058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.040058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.040183 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.040271 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.040594 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.040665 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 08:18:39.041041 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.030280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:44947 (local address 127.0.0.1:43014)
0620 08:18:39.030468 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:39.030505 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:39.031076 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:39.031385 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:39.031394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:39.031822 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:39.032138 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.032143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.033048 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.033048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.034154 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.034156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.034605 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.034606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.034710 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.035225 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:39.035235 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:39.037056 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:39.039415 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.039421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.039430 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.039631 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.039865 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.039866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.039866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.039985 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.040262 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:39.040269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:39.040438 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 08:18:39.040560 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 08:18:39.040735 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260620 08:18:39.041631 15550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:39.041720 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.031727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.24.249.188:38825)
0620 08:18:39.031879 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.031883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.031905 (+ 22us) server_negotiation.cc:407] Connection header received
0620 08:18:39.031970 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.031972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.031991 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.032078 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.032836 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.034101 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.035104 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.035566 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.035649 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.038454 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.038464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.038465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.038472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.040363 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.040761 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.040764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.040764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.040831 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.041077 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.041078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.041078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.041267 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.041436 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.041486 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.041580 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943519, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:39.049337 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.041983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:42239 (local address 127.0.0.1:60826)
0620 08:18:39.042022 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:39.042031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:39.042065 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:39.042222 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:39.042223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:39.042364 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:39.042424 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.042425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.043067 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.043069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.043721 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.043724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.044123 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.044124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.044199 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.044636 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:39.044641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:39.046115 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:39.048215 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.048216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.048217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.048440 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.048693 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.048693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.048694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.048748 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.049058 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:39.049059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:39.049110 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 08:18:39.049183 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:18:39.049267 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 08:18:39.049436 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.042051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.24.249.189:42239)
0620 08:18:39.042097 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.042099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.042107 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:39.042132 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.042133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.042149 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.042196 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.042504 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.042975 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.043808 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.044050 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.044102 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.046223 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.046225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.046225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.046233 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.047923 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.048530 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.048531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.048531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.048616 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.048814 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.048815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.048816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.048959 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.049047 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.049229 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.049310 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:18:39.049829 15550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:39.051769 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.043214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.24.249.190:44947)
0620 08:18:39.043258 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.043260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.043269 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:39.043317 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.043318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.043328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.043393 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.043841 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.044411 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.045572 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.045812 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.045943 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.048609 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.048611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.048611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.048620 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.050498 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.050928 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.050929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.050930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.050993 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.051180 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.051181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.051181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.051362 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.051564 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.051606 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.051671 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943519, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:18:39.057732 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.050149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47508)
0620 08:18:39.050202 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:39.050213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:39.050250 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:39.050442 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:39.050443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:39.050613 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:39.050659 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.050661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.051525 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.051526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.052577 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.052580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.053053 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.053054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.053118 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.053500 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:39.053505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:39.054777 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:39.056498 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.056499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.056499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.056755 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.057033 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.057034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.057035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.057115 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.057396 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:39.057398 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:39.057436 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 08:18:39.057522 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 08:18:39.057621 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:18:39.057787 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.050241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.24.249.188:38825)
0620 08:18:39.050283 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.050285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.050293 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:39.050315 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.050316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.050327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.050396 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.050750 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.051444 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.052659 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.052980 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.053042 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.054889 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.054890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.054890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.054896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.056413 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.056853 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.056854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.056854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.056908 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.057190 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.057191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.057191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.057315 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.057393 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.057544 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.057668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":25}
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:18:39.060324 15550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: Bootstrap starting.
I20260620 08:18:39.061339 15550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:39.061676 15550 log.cc:824] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:39.062284 15400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } attempt: 2
I20260620 08:18:39.062681 15550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: No bootstrap required, opened a new log
I20260620 08:18:39.063455 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.052792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.24.249.189:42239)
0620 08:18:39.052832 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.052834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.052841 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:18:39.052861 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.052862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.052872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.052928 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.053540 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.054130 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.054847 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.055756 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.055815 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.058993 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.058995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.058995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.059002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.060614 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.062381 (+ 1767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.062383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.062383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.062450 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.062695 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.062696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.062697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.062871 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.063009 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.063276 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.063361 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:39.065181 15550 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.065358 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:39.065371 15550 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f386c8bfe1474e8971a821772e7aa0, State: Initialized, Role: FOLLOWER
I20260620 08:18:39.065469 15550 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.065744 15556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.065807 15550 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:39.065832 15556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:18:39.066859 15374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "21324ad537c84f288dc40977ce254d8e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43028
W20260620 08:18:39.067749 15567 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:39.068295 15567 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:39.070887 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.062555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47510)
0620 08:18:39.062606 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:39.062619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:39.062660 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:39.062859 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:39.062860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:39.063018 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:39.063062 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.063064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.063993 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.063994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.064977 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.064981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.065406 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.065407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.065463 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.066012 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:39.066017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:39.067572 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:39.069537 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.069538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.069539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.069801 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.070084 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.070085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.070085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.070165 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.070530 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:39.070531 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:39.070599 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 08:18:39.070671 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 08:18:39.070792 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 08:18:39.070996 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.062674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.24.249.188:38825)
0620 08:18:39.062714 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.062716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.062724 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:39.062746 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.062746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.062756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.062844 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.063171 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.063921 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.065089 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.065292 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.065625 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.067669 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.067670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.067671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.067692 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.069420 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.069932 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.069934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.069934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.070015 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.070247 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.070248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.070248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.070419 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.070523 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.070703 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.070846 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:18:39.068837 15482 master_runner.cc:186] Error getting master registration for 127.24.249.190:44947: OK, instance_id { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" instance_seqno: 1781943518715817 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:18:39.072273 15482 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:18:39.073062 15400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e: Bootstrap starting.
I20260620 08:18:39.073624 15400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:39.073982 15400 log.cc:824] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:39.074887 15400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e: No bootstrap required, opened a new log
I20260620 08:18:39.076354 15400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.076499 15400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21324ad537c84f288dc40977ce254d8e, State: Initialized, Role: FOLLOWER
I20260620 08:18:39.076586 15400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.076869 15570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.076934 15400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:39.077005 15570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:39.077008 15374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 08:18:39.077183 15374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:39.078142 15374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.078388 15374 leader_election.cc:288] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [CANDIDATE]: Term 1 election: Requested vote from peers 5bc13c2efe834c0498d10a57752dd0a3 (127.24.249.189:42239), c2f386c8bfe1474e8971a821772e7aa0 (127.24.249.188:38825)
I20260620 08:18:39.078724 15524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21324ad537c84f288dc40977ce254d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2f386c8bfe1474e8971a821772e7aa0"
I20260620 08:18:39.078860 15524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [term 0 FOLLOWER]: Advancing to term 1
W20260620 08:18:39.078970 15581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:39.079051 15581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:39.079741 15524 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21324ad537c84f288dc40977ce254d8e in term 1.
I20260620 08:18:39.079982 15351 leader_election.cc:302] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21324ad537c84f288dc40977ce254d8e, c2f386c8bfe1474e8971a821772e7aa0; no voters:
I20260620 08:18:39.080103 15570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:39.080237 15570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [term 1 LEADER]: Becoming Leader. State: Replica: 21324ad537c84f288dc40977ce254d8e, State: Running, Role: LEADER
I20260620 08:18:39.080328 15570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.080751 15583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21324ad537c84f288dc40977ce254d8e. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.080895 15583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:39.083179 15332 master_runner.cc:186] Error getting master registration for 127.24.249.189:42239: OK, instance_id { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" instance_seqno: 1781943518871128 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:18:39.086170 15332 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:18:39.090399 15585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:39.090665 15585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:39.092451 15524 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [term 1 FOLLOWER]: Refusing update from remote peer 21324ad537c84f288dc40977ce254d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:39.092760 15583 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:39.094058 15556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21324ad537c84f288dc40977ce254d8e. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.094149 15556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:39.096935 15583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.097023 15583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:39.097482 15556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.097568 15556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:39.097810 15593 mvcc.cc:205] Tried to move back new op lower bound from 7298840654199853056 to 7298840654153723904. Current Snapshot: MvccSnapshot[applied={T|T < 7298840654199853056}]
I20260620 08:18:39.098373 15594 mvcc.cc:205] Tried to move back new op lower bound from 7298840654199853056 to 7298840654153723904. Current Snapshot: MvccSnapshot[applied={T|T < 7298840654199853056}]
I20260620 08:18:39.099061 15585 catalog_manager.cc:1347] Generated new cluster ID: b2011c48da454aaaa9dca94af383e2b7
I20260620 08:18:39.099110 15585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:39.099982 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.086494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.24.249.188:38825)
0620 08:18:39.086540 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.086542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.089478 (+ 2936us) server_negotiation.cc:407] Connection header received
0620 08:18:39.089549 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.089551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.089562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.089622 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.091540 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.092171 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.093146 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.093454 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.093514 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.096854 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.096856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.096856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.096863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.098559 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.098944 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.098945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.098946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.099001 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.099204 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.099205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.099205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.099335 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.099436 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.099813 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.099889 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 08:18:39.101980 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.082766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.24.249.190:44947)
0620 08:18:39.082817 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.082819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.086723 (+ 3904us) server_negotiation.cc:407] Connection header received
0620 08:18:39.086831 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.086832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.086842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.086873 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.087620 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.088359 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.089587 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.089915 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.089961 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.094371 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.094372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.094373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.094380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.096280 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.101052 (+ 4772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.101053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.101054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.101118 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.101331 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.101332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.101332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.101468 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.101579 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.101791 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.101887 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 08:18:39.106658 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.082685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.24.249.189:42239)
0620 08:18:39.082730 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.082733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.086816 (+ 4083us) server_negotiation.cc:407] Connection header received
0620 08:18:39.086951 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.086953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.086976 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.087007 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.088126 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.088880 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.094687 (+ 5807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.095010 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.095052 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.103638 (+ 8586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.103640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.103641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.103649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.105365 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.105763 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.105764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.105765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.105821 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.106048 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.106049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.106049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.106183 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.106275 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.106478 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.106561 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:39.107577 15475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } attempt: 3
Jun 20 08:18:39 dist-test-slave-65w8 krb5kdc[15319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943518, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:18:39.119220 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.107824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:38825 (local address 127.0.0.1:47520)
0620 08:18:39.107867 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:39.107876 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:39.107910 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:39.108061 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:39.108062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:39.108222 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:39.108266 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.108268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.109223 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.109224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.110105 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:39.110107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.110629 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.110630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:39.110713 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.111461 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:39.111466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:39.112985 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:39.117520 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.117522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.117523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.117776 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.118139 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:39.118140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:39.118140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:39.118215 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:39.118901 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:39.118902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:39.118953 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 08:18:39.119000 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 08:18:39.119113 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 08:18:39.119869 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:39.107875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.24.249.188:38825)
0620 08:18:39.107933 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.107935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.107946 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:18:39.107970 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.107971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.107981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.108055 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.108325 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.109136 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.110191 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:39.110522 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:39.110877 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:39.113096 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:39.113098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:39.113098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:39.113105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:39.117357 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.117917 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.117918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.117918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.118037 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:39.118474 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:39.118475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:39.118475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:39.118775 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:39.119472 (+ 697us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:39.119544 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 08:18:39.119683 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:18:39.121507 15475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: Bootstrap starting.
I20260620 08:18:39.122294 15475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:39.122601 15475 log.cc:824] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:39.134685 15475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: No bootstrap required, opened a new log
I20260620 08:18:39.137072 15475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.137382 15475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc13c2efe834c0498d10a57752dd0a3, State: Initialized, Role: FOLLOWER
I20260620 08:18:39.137534 15475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } }
I20260620 08:18:39.139329 15449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21324ad537c84f288dc40977ce254d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5bc13c2efe834c0498d10a57752dd0a3"
I20260620 08:18:39.139765 15448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:39.140772 15448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [term 1 FOLLOWER]: Refusing update from remote peer 21324ad537c84f288dc40977ce254d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:39.141113 15570 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:39.142191 15475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:39.143646 15599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.143775 15599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:39.143847 15599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21324ad537c84f288dc40977ce254d8e. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.143900 15599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:39.143935 15599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.143986 15599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:39.144049 15585 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:39.144933 15585 catalog_manager.cc:1504] Loading token signing keys...
W20260620 08:18:39.150832 15615 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:39.150904 15615 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:39.150950 15604 mvcc.cc:205] Tried to move back new op lower bound from 7298840654199853056 to 7298840654153723904. Current Snapshot: MvccSnapshot[applied={T|T < 7298840654199853056}]
I20260620 08:18:39.151131 15449 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 21324ad537c84f288dc40977ce254d8e for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 08:18:39.151350 15570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21324ad537c84f288dc40977ce254d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21324ad537c84f288dc40977ce254d8e" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 44947 } } peers { permanent_uuid: "5bc13c2efe834c0498d10a57752dd0a3" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 42239 } } peers { permanent_uuid: "c2f386c8bfe1474e8971a821772e7aa0" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 38825 } } }
I20260620 08:18:39.151439 15570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:39.163530 15585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21324ad537c84f288dc40977ce254d8e: Generated new TSK 0
I20260620 08:18:39.163772 15585 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:18:39.184755 15554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.183651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.24.249.188:38825)
0620 08:18:39.183710 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.183713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.184049 (+ 336us) server_negotiation.cc:407] Connection header received
0620 08:18:39.184160 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.184162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.184173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.184206 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.184680 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47522: BlockingRecv error: recv got EOF from 127.0.0.1:47522 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 08:18:39.186196 15403 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.185290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.24.249.190:44947)
0620 08:18:39.185332 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.185334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.185612 (+ 278us) server_negotiation.cc:407] Connection header received
0620 08:18:39.185727 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.185728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.185740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.185770 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.186127 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43042: BlockingRecv error: recv got EOF from 127.0.0.1:43042 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 08:18:39.187160 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.185502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.24.249.189:42239)
0620 08:18:39.185540 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.185542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.186618 (+ 1076us) server_negotiation.cc:407] Connection header received
0620 08:18:39.186741 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.186742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.186754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.186785 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.187097 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60864: BlockingRecv error: recv got EOF from 127.0.0.1:60864 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 08:18:39.191057 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.190161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.24.249.189:42239)
0620 08:18:39.190203 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.190205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.190475 (+ 270us) server_negotiation.cc:407] Connection header received
0620 08:18:39.190591 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.190592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.190612 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.190645 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.191005 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60878: BlockingRecv error: recv got EOF from 127.0.0.1:60878 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 08:18:39.194408 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.193521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.24.249.189:42239)
0620 08:18:39.193558 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.193559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.193808 (+ 249us) server_negotiation.cc:407] Connection header received
0620 08:18:39.193943 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.193945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.193957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.193989 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.194358 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60894: BlockingRecv error: recv got EOF from 127.0.0.1:60894 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 08:18:39.198679 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.197783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.24.249.189:42239)
0620 08:18:39.197819 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.197821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.198118 (+ 297us) server_negotiation.cc:407] Connection header received
0620 08:18:39.198230 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.198231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.198243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.198275 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.198613 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60898: BlockingRecv error: recv got EOF from 127.0.0.1:60898 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 08:18:39.205062 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.204182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.24.249.189:42239)
0620 08:18:39.204221 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.204223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.204457 (+ 234us) server_negotiation.cc:407] Connection header received
0620 08:18:39.204563 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.204564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.204575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.204606 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.205011 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60904: BlockingRecv error: recv got EOF from 127.0.0.1:60904 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 08:18:39.215595 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.214606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.24.249.189:42239)
0620 08:18:39.214676 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.214678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.214969 (+ 291us) server_negotiation.cc:407] Connection header received
0620 08:18:39.215088 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.215089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.215101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.215135 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.215538 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60908: BlockingRecv error: recv got EOF from 127.0.0.1:60908 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 08:18:39.234357 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.233293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.24.249.189:42239)
0620 08:18:39.233345 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.233347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.233619 (+ 272us) server_negotiation.cc:407] Connection header received
0620 08:18:39.233762 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.233763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.233777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.233812 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.234289 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60920: BlockingRecv error: recv got EOF from 127.0.0.1:60920 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 08:18:39.268785 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.268016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.24.249.189:42239)
0620 08:18:39.268061 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.268063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.268170 (+ 107us) server_negotiation.cc:407] Connection header received
0620 08:18:39.268226 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.268228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.268240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.268322 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.268724 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60934: BlockingRecv error: recv got EOF from 127.0.0.1:60934 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 08:18:39.334616 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.334023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.24.249.189:42239)
0620 08:18:39.334087 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.334089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.334128 (+ 39us) server_negotiation.cc:407] Connection header received
0620 08:18:39.334180 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.334181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.334193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.334262 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.334567 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60936: BlockingRecv error: recv got EOF from 127.0.0.1:60936 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 08:18:39.464773 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.463879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.24.249.189:42239)
0620 08:18:39.463944 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.463947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.464084 (+ 137us) server_negotiation.cc:407] Connection header received
0620 08:18:39.464151 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.464153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.464169 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.464237 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.464694 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60948: BlockingRecv error: recv got EOF from 127.0.0.1:60948 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 08:18:39.722837 15479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:39.722003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.24.249.189:42239)
0620 08:18:39.722188 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:39.722191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:39.722202 (+ 11us) server_negotiation.cc:407] Connection header received
0620 08:18:39.722341 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:39.722342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:39.722355 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:39.722432 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:39.722786 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60964: BlockingRecv error: recv got EOF from 127.0.0.1:60964 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260620 08:18:40.068871 15567 catalog_manager.cc:1259] Loaded cluster ID: b2011c48da454aaaa9dca94af383e2b7
I20260620 08:18:40.068969 15567 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: loading cluster ID for follower catalog manager: success
I20260620 08:18:40.069779 15567 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: acquiring CA information for follower catalog manager: success
I20260620 08:18:40.070123 15567 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2f386c8bfe1474e8971a821772e7aa0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:18:40.151360 15615 catalog_manager.cc:1259] Loaded cluster ID: b2011c48da454aaaa9dca94af383e2b7
I20260620 08:18:40.151453 15615 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: loading cluster ID for follower catalog manager: success
I20260620 08:18:40.152289 15615 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: acquiring CA information for follower catalog manager: success
I20260620 08:18:40.152633 15615 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5bc13c2efe834c0498d10a57752dd0a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:18:40.238919 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:40.236054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.24.249.189:42239)
0620 08:18:40.236207 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:40.236210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:40.236290 (+ 80us) server_negotiation.cc:407] Connection header received
0620 08:18:40.236439 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:40.236441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:40.236497 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:40.236585 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:40.236769 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.237308 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.238106 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.238308 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.238384 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:40.238565 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:40.238675 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:40.238769 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:40.238788 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 08:18:40.238812 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260620 08:18:40.243781 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:40.240799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.24.249.188:38825)
0620 08:18:40.240903 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:40.240905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:40.240970 (+ 65us) server_negotiation.cc:407] Connection header received
0620 08:18:40.241038 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:40.241038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:40.241074 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:40.241145 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:40.241231 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.241715 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.242653 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.242886 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.242960 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:40.243273 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:40.243499 (+ 226us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:40.243626 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:40.243649 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 08:18:40.243674 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260620 08:18:40.249019 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:40.245926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.24.249.188:38825)
0620 08:18:40.245969 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:40.245972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:40.246120 (+ 148us) server_negotiation.cc:407] Connection header received
0620 08:18:40.246176 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:40.246177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:40.246190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:40.246248 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:40.246409 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.247173 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.248062 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.248254 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.248336 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:40.248499 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:40.248552 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:40.248708 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:40.248899 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 08:18:40.248925 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:40.249425 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:40.246300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.24.249.189:42239)
0620 08:18:40.246350 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:40.246353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:40.246597 (+ 244us) server_negotiation.cc:407] Connection header received
0620 08:18:40.246692 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:40.246693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:40.246710 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:40.246770 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:40.246913 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.247500 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.248596 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.248819 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.248898 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:40.249136 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:40.249173 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:40.249205 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:40.249330 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 08:18:40.249355 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 08:18:40.249687 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:40.245908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.24.249.190:44947)
0620 08:18:40.246029 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:40.246031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:40.246190 (+ 159us) server_negotiation.cc:407] Connection header received
0620 08:18:40.246314 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:40.246315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:40.246380 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:40.246441 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:40.246692 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.247406 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.248572 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:40.248996 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:40.249067 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:40.249290 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:40.249408 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:40.249451 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:40.249577 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 08:18:40.249601 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260620 08:18:40.251547 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15332
I20260620 08:18:40.257256 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15407
I20260620 08:18:40.262559 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15482
2026-06-20T08:18:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2890 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 08:18:40 dist-test-slave-65w8 krb5kdc[15698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 08:18:40 dist-test-slave-65w8 krb5kdc[15698](info): set up 2 sockets
Jun 20 08:18:40 dist-test-slave-65w8 krb5kdc[15698](info): commencing operation
krb5kdc: starting...
W20260620 08:18:41.314457 25574 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/test-user.keytab.
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T08:18:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T08:18:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.249.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
Entry for principal kudu/127.24.249.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab.
I20260620 08:18:41.437748 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34615
--webserver_interface=127.24.249.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34615,127.24.249.189:38497,127.24.249.188:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 08:18:41.520977 15711 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:18:41.521235 15711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:41.521266 15711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:41.521296 15711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:41.522498 15711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:41.522567 15711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:41.522590 15711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:41.522619 15711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:41.522648 15711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:41.524154 15711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34615,127.24.249.189:38497,127.24.249.188:45907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.190.keytab
--principal=kudu/127.24.249.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.190:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.249.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:41.524441 15711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:41.524667 15711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:41.530210 15719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:41.530229 15716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:41.530246 15717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:41.530274 15711 server_base.cc:1034] running on GCE node
I20260620 08:18:41.530714 15711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:41.531044 15711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:41.532213 15711 hybrid_clock.cc:648] HybridClock initialized: now 1781943521532193 us; error 26 us; skew 500 ppm
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:41.534801 15711 init.cc:377] Logged in from keytab as kudu/127.24.249.190@KRBTEST.COM (short username kudu)
I20260620 08:18:41.535276 15711 webserver.cc:460] Webserver started at http://127.24.249.190:39683/ using document root <none> and password file <none>
I20260620 08:18:41.535508 15711 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:41.535573 15711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:41.535722 15711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:41.536773 15711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data/instance:
uuid: "75404e021bf34883abbce8a9c6470523"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "f995bc72d4a2ecdd9a99aa7454f3e1fb"
server_key_iv: "8d1feb54e41812d4ef2f78ec244e4613"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.537137 15711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/wal/instance:
uuid: "75404e021bf34883abbce8a9c6470523"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "f995bc72d4a2ecdd9a99aa7454f3e1fb"
server_key_iv: "8d1feb54e41812d4ef2f78ec244e4613"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.538605 15711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 08:18:41.539494 15726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:41.539698 15711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:41.539795 15711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/wal
uuid: "75404e021bf34883abbce8a9c6470523"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "f995bc72d4a2ecdd9a99aa7454f3e1fb"
server_key_iv: "8d1feb54e41812d4ef2f78ec244e4613"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.539877 15711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:41.556771 15711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:41.559139 15711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:41.559363 15711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:41.563568 15711 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.190:34615
I20260620 08:18:41.563606 15778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.190:34615 every 8 connection(s)
I20260620 08:18:41.563964 15711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/data/info.pb
I20260620 08:18:41.564867 15779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:41.566527 15779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:41.572904 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15711
I20260620 08:18:41.573009 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-0/wal/instance
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:41.573222 25574 external_mini_cluster.cc:1428] Setting key d3bf9658fe88c6f7b0b3805e7ed9cbd1
I20260620 08:18:41.577359 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.567317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.24.249.190:34615)
0620 08:18:41.567430 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.567435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.567471 (+ 36us) server_negotiation.cc:407] Connection header received
0620 08:18:41.568176 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.568178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.568194 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.568274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.569034 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.569890 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.570808 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.571124 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.571206 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.573606 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.573613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.573614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.573620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.575744 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.576252 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.576254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.576255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.576335 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.576585 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.576586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.576586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.576796 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.576878 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.577073 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.577197 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260620 08:18:41.577518 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.567159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34615 (local address 127.0.0.1:51644)
0620 08:18:41.567318 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.567373 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.568041 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.568323 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.568326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.568775 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.568942 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.568947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.570014 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.570015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.570716 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.570719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.571234 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.571235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.571301 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.571824 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.571834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.573492 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.575855 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.575862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.575863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.576172 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.576404 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.576405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.576405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.576520 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.576861 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.576867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.576945 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.577050 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.577288 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":63,"thread_start_us":66,"threads_started":1}
I20260620 08:18:41.578496 15779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:41.578871 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.578728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:46984)
0620 08:18:41.578781 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.578837 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 08:18:41.579025 15730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
I20260620 08:18:41.579474 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.579385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:46998)
0620 08:18:41.579429 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.579456 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 08:18:41.579597 15779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:38497: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.249.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab.
Entry for principal kudu/127.24.249.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab.
I20260620 08:18:41.602049 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:38497
--webserver_interface=127.24.249.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab
--principal=kudu/127.24.249.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34615,127.24.249.189:38497,127.24.249.188:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:18:41.628854 15779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } attempt: 1
I20260620 08:18:41.629472 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.629307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:47010)
0620 08:18:41.629397 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.629442 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 08:18:41.629930 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.629793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:47018)
0620 08:18:41.629836 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.629899 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 08:18:41.630028 15779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:38497: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
W20260620 08:18:41.686848 15786 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:18:41.687079 15786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:41.687109 15786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:41.687165 15786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:41.688405 15786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:41.688473 15786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:41.688500 15786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:41.688527 15786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:41.688553 15786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:41.690145 15786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34615,127.24.249.189:38497,127.24.249.188:45907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.189.keytab
--principal=kudu/127.24.249.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.189:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.249.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:41.690423 15786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:41.690649 15786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:41.695488 15794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:41.695497 15792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:41.695488 15791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:41.695724 15786 server_base.cc:1034] running on GCE node
I20260620 08:18:41.695977 15786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:41.696281 15786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:41.697435 15786 hybrid_clock.cc:648] HybridClock initialized: now 1781943521697410 us; error 37 us; skew 500 ppm
I20260620 08:18:41.698176 15779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } attempt: 2
I20260620 08:18:41.698655 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.698494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:47026)
0620 08:18:41.698568 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.698611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 08:18:41.699150 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.699017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:47038)
0620 08:18:41.699095 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.699126 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 08:18:41.699323 15779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.189:38497: Network error: Client connection negotiation failed: client connection to 127.24.249.189:38497: connect: Connection refused (error 111)
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:41.700551 15786 init.cc:377] Logged in from keytab as kudu/127.24.249.189@KRBTEST.COM (short username kudu)
I20260620 08:18:41.701025 15786 webserver.cc:460] Webserver started at http://127.24.249.189:44051/ using document root <none> and password file <none>
I20260620 08:18:41.701280 15786 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:41.701335 15786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:41.701509 15786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:41.702766 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data/instance:
uuid: "ec6a4b76874042549af2904f1b0c6f8e"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "fdcccb5dd0bdde7734c0d28ff80617b5"
server_key_iv: "f7cfa4b01d74b5b22a91c80058cbe582"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.703140 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/wal/instance:
uuid: "ec6a4b76874042549af2904f1b0c6f8e"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "fdcccb5dd0bdde7734c0d28ff80617b5"
server_key_iv: "f7cfa4b01d74b5b22a91c80058cbe582"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.704615 15786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 08:18:41.705482 15801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:41.705705 15786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 08:18:41.705817 15786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/wal
uuid: "ec6a4b76874042549af2904f1b0c6f8e"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "fdcccb5dd0bdde7734c0d28ff80617b5"
server_key_iv: "f7cfa4b01d74b5b22a91c80058cbe582"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.705956 15786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:41.736397 15786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:41.739151 15786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:41.739461 15786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:41.744357 15786 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.189:38497
I20260620 08:18:41.744405 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.189:38497 every 8 connection(s)
I20260620 08:18:41.744786 15786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/data/info.pb
I20260620 08:18:41.745730 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:41.747671 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15786
I20260620 08:18:41.747776 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-1/wal/instance
I20260620 08:18:41.747975 25574 external_mini_cluster.cc:1428] Setting key d7e6e177fa97f45d1eeaf8a5d22c3d9f
I20260620 08:18:41.747893 15854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.249.188@KRBTEST.COM; defaulting to no policy
I20260620 08:18:41.759209 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.748923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.24.249.190:34615)
0620 08:18:41.749001 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.749003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.749260 (+ 257us) server_negotiation.cc:407] Connection header received
0620 08:18:41.749979 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.749981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.749998 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.750069 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.750774 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.751317 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.752916 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.753126 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.753202 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.755820 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.755822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.755822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.755831 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.757586 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.758143 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.758144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.758144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.758197 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.758437 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.758438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.758438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.758564 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.758622 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.759070 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.759143 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:18:41.759404 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.748903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34615 (local address 127.0.0.1:51652)
0620 08:18:41.749113 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.749163 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.749829 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.750144 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.750147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.750405 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.750627 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.750632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.751421 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.751423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.752799 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.752803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.753222 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.753223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.753396 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.754062 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.754074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.755723 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.757692 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.757699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.757701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.758086 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.758244 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.758245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.758245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.758337 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.758668 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.758675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.758865 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.759006 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.759136 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.249.188@KRBTEST.COM" created.
I20260620 08:18:41.760097 15854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:18:41.768198 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.760381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:47050)
0620 08:18:41.760433 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.760443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.760491 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.760756 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.760757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.760870 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.760923 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.760924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.762072 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.762074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.762938 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.762942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.763468 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.763469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.763554 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.764022 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.764028 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.765361 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.767161 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.767162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.767163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.767330 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.767538 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.767539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.767540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.767594 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.767873 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.767874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.767911 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.767982 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.768081 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:18:41.768265 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.760431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.24.249.189:38497)
0620 08:18:41.760598 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.760603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.760627 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:18:41.760651 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.760653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.760668 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.760749 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.760991 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.761989 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.763035 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.763383 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.763449 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.765457 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.765466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.765466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.765473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.767063 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.767419 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.767422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.767422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.767476 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.767659 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.767660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.767661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.767806 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.767895 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.768013 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.768134 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260620 08:18:41.768947 15854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:41.769397 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.769220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:43994)
0620 08:18:41.769315 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.769365 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 08:18:41.769595 15802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
I20260620 08:18:41.770164 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.770032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44006)
0620 08:18:41.770086 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.770121 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 08:18:41.770318 15854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45907: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.249.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab.
Entry for principal kudu/127.24.249.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab.
I20260620 08:18:41.776957 25574 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5EkY3n/build/release/bin/kudu
/tmp/dist-test-task5EkY3n/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:45907
--webserver_interface=127.24.249.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab
--principal=kudu/127.24.249.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.249.148:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.249.190:34615,127.24.249.189:38497,127.24.249.188:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 08:18:41.790480 15779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } attempt: 3
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:18:41.798593 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.790946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:47056)
0620 08:18:41.791054 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.791067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.791118 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.791292 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.791294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.791456 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.791521 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.791523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.792504 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.792505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.793037 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.793039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.793539 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.793540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.793660 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.794194 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.794199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.795673 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.797573 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.797574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.797574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.797765 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.797996 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.797997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.797997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.798066 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.798329 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.798330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.798376 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.798436 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.798520 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 08:18:41.798648 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.791055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.24.249.189:38497)
0620 08:18:41.791127 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.791130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.791142 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:18:41.791176 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.791177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.791193 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.791284 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.791600 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.792401 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.793118 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.793455 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.793526 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.795766 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.795768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.795769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.795777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.797482 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.797842 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.797843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.797844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.797924 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.798122 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.798123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.798123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.798261 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.798327 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.798474 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.798557 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 08:18:41.799216 15779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:41.799580 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.799464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44012)
0620 08:18:41.799513 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.799550 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 08:18:41.799906 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.799819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44028)
0620 08:18:41.799861 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.799887 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 08:18:41.800000 15779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45907: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
I20260620 08:18:41.819564 15854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } attempt: 1
I20260620 08:18:41.820040 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.819912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44040)
0620 08:18:41.819971 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.820009 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 08:18:41.820405 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.820300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44056)
0620 08:18:41.820342 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.820386 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 08:18:41.820490 15854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45907: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
I20260620 08:18:41.831162 15779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } attempt: 1
I20260620 08:18:41.831657 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.831511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44066)
0620 08:18:41.831578 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.831624 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 08:18:41.831997 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.831906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44078)
0620 08:18:41.831946 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.831975 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 08:18:41.832091 15779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45907: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
W20260620 08:18:41.862775 15861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 08:18:41.862995 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 08:18:41.863013 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 08:18:41.863034 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 08:18:41.864240 15861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 08:18:41.864282 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 08:18:41.864296 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 08:18:41.864310 15861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 08:18:41.864321 15861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 08:18:41.865809 15861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.249.148:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.249.190:34615,127.24.249.189:38497,127.24.249.188:45907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/krb5kdc/kudu_127.24.249.188.keytab
--principal=kudu/127.24.249.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.249.188:45907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.249.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 06:58:15 UTC on ccycloud.quasar-vppxbf.root.comops.site
build id 4457017
I20260620 08:18:41.866030 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 08:18:41.866209 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 08:18:41.871012 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:41.870992 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 08:18:41.870992 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 08:18:41.871232 15861 server_base.cc:1034] running on GCE node
I20260620 08:18:41.871534 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 08:18:41.871822 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 08:18:41.872997 15861 hybrid_clock.cc:648] HybridClock initialized: now 1781943521872982 us; error 45 us; skew 500 ppm
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 08:18:41.875509 15861 init.cc:377] Logged in from keytab as kudu/127.24.249.188@KRBTEST.COM (short username kudu)
I20260620 08:18:41.875957 15861 webserver.cc:460] Webserver started at http://127.24.249.188:34717/ using document root <none> and password file <none>
I20260620 08:18:41.876160 15861 fs_manager.cc:359] Metadata directory not provided
I20260620 08:18:41.876201 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 08:18:41.876289 15861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 08:18:41.877322 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data/instance:
uuid: "13043fa9fd504f4fbe7b5f97db7b2f10"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "8fb5d2d5ed4de349af73f5afbcc2e33d"
server_key_iv: "7dc5cf4181474f9589f45992032ba9a9"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.877635 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/wal/instance:
uuid: "13043fa9fd504f4fbe7b5f97db7b2f10"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "8fb5d2d5ed4de349af73f5afbcc2e33d"
server_key_iv: "7dc5cf4181474f9589f45992032ba9a9"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.879082 15861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 08:18:41.879889 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 08:18:41.880084 15861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 08:18:41.880177 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data,/tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/wal
uuid: "13043fa9fd504f4fbe7b5f97db7b2f10"
format_stamp: "Formatted at 2026-06-20 08:18:41 on dist-test-slave-65w8"
server_key: "8fb5d2d5ed4de349af73f5afbcc2e33d"
server_key_iv: "7dc5cf4181474f9589f45992032ba9a9"
server_key_version: "encryptionkey@0"
I20260620 08:18:41.880237 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 08:18:41.888633 15854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } attempt: 2
I20260620 08:18:41.889104 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.888977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44082)
0620 08:18:41.889035 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.889074 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 08:18:41.889518 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.889426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44098)
0620 08:18:41.889470 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.889499 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 08:18:41.889653 15854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45907: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
I20260620 08:18:41.907266 15779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } attempt: 2
I20260620 08:18:41.907809 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.907647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44104)
0620 08:18:41.907717 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.907768 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 08:18:41.908236 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.908127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44120)
0620 08:18:41.908167 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.908191 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 08:18:41.908319 15779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.249.188:45907: Network error: Client connection negotiation failed: client connection to 127.24.249.188:45907: connect: Connection refused (error 111)
I20260620 08:18:41.914139 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 08:18:41.916527 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 08:18:41.916646 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 08:18:41.920564 15861 rpc_server.cc:307] RPC server started. Bound to: 127.24.249.188:45907
I20260620 08:18:41.920930 15928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.249.188:45907 every 8 connection(s)
I20260620 08:18:41.920969 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/data/info.pb
I20260620 08:18:41.921787 15929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 08:18:41.922513 25574 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5EkY3n/build/release/bin/kudu as pid 15861
I20260620 08:18:41.922616 25574 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5EkY3n/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781943243671421-25574-0/minicluster-data/master-2/wal/instance
I20260620 08:18:41.922824 25574 external_mini_cluster.cc:1428] Setting key a59ff8ffc767c9638559df8596e8c917
I20260620 08:18:41.923885 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:18:41.934202 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.924809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.24.249.190:34615)
0620 08:18:41.924862 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.924864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.925031 (+ 167us) server_negotiation.cc:407] Connection header received
0620 08:18:41.925088 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.925089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.925103 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.925179 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.925635 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.926432 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.927824 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.928107 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.928179 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.930842 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.930843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.930843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.930850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.932549 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.933003 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.933004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.933004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.933058 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.933347 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.933348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.933349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.933503 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.933579 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.934056 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.934132 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 08:18:41.934525 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.924714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.190:34615 (local address 127.0.0.1:51662)
0620 08:18:41.924895 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.924940 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.925017 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.925205 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.925207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.925458 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.925548 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.925552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.926664 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.926665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.927749 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.927752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.928234 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.928235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.928311 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.928937 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.928947 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.930736 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.932640 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.932646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.932649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.932920 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.933126 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.933127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.933128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.933270 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.933584 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.933590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.933897 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.934001 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.934173 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260620 08:18:41.934906 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.923391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.24.249.188:45907)
0620 08:18:41.923582 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.923592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.923623 (+ 31us) server_negotiation.cc:407] Connection header received
0620 08:18:41.924054 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.924061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.924108 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.924255 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.925125 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.926501 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.927674 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.928100 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.928219 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.931342 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.931350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.931350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.931363 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.933268 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.933745 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.933748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.933749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.933807 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.934065 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.934066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.934067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.934225 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.934379 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.934599 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.934697 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260620 08:18:41.935072 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.190@KRBTEST.COM
I20260620 08:18:41.942829 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.935501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.24.249.189:38497)
0620 08:18:41.935545 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.935547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.935555 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:41.935580 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.935581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.935592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.935647 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.935991 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.936762 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.937762 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.938053 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.938138 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.940025 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.940026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.940027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.940035 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.941595 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.942016 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.942017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.942018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.942079 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.942259 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.942260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.942261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.942399 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.942505 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.942663 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.942737 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 08:18:41.942867 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.935394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.189:38497 (local address 127.0.0.1:47072)
0620 08:18:41.935429 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.935439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.935482 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.935659 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.935660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.935854 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.935913 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.935915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.936873 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.936874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.937680 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.937682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.938135 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.938136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.938206 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.938658 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.938663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.939933 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.941705 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.941707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.941707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.941930 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.942141 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.942141 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.942142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.942202 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.942496 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.942496 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.942552 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.942635 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.942734 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 08:18:41.943244 15929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } has no permanent_uuid. Determining permanent_uuid...
I20260620 08:18:41.943593 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.935712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.24.249.190:34615)
0620 08:18:41.935748 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.935750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.935757 (+ 7us) server_negotiation.cc:407] Connection header received
0620 08:18:41.935775 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.935775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.935784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.935844 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.936463 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.937150 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.938043 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.938257 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.938337 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.940498 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.940500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.940500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.940508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.942074 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.942449 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.942450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.942451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.942504 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.942700 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.942701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.942702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.942846 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.943045 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.943428 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.943491 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.188@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.249.189@KRBTEST.COM
I20260620 08:18:41.950031 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.943601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44136)
0620 08:18:41.943667 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.943677 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.943720 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.943832 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.943832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.943959 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.943984 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.943985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.944699 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.944700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.945284 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.945286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.945695 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.945696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.945749 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.946179 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.946184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.947318 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.948866 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.948867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.948867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.949092 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.949332 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.949332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.949333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.949384 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.949662 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.949663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.949743 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.949814 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.949968 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 08:18:41.950135 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.943636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.24.249.188:45907)
0620 08:18:41.943670 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.943672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.943706 (+ 34us) server_negotiation.cc:407] Connection header received
0620 08:18:41.943774 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.943775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.943784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.943830 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.944072 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.944625 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.945354 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.945631 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.945679 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.947408 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.947410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.947410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.947416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.948779 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.949183 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.949184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.949185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.949249 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.949443 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.949444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.949445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.949589 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.949654 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.949906 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.950031 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":16}
I20260620 08:18:41.951491 15929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: Bootstrap starting.
I20260620 08:18:41.951758 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.944291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.24.249.189:38497)
0620 08:18:41.944330 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.944332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.944340 (+ 8us) server_negotiation.cc:407] Connection header received
0620 08:18:41.944378 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.944379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.944388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.944442 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.944947 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.945428 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.946242 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.946433 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.946500 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.948772 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.948773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.948774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.948781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.950635 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.950973 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.950974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.950975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.951029 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.951235 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.951236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.951236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.951347 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.951533 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.951569 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.951641 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:18:41.952617 15929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:41.952945 15929 log.cc:824] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:41.953141 25574 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 08:18:41.953734 15929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: No bootstrap required, opened a new log
I20260620 08:18:41.954388 15753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "75404e021bf34883abbce8a9c6470523"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51672
I20260620 08:18:41.955281 15929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:41.955415 15929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13043fa9fd504f4fbe7b5f97db7b2f10, State: Initialized, Role: FOLLOWER
I20260620 08:18:41.955503 15929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:41.955813 15929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:41.955792 15935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:41.955888 15935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:18:41.957684 15946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:41.957755 15946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:41.958077 15861 master_runner.cc:186] Error getting master registration for 127.24.249.190:34615: OK, instance_id { permanent_uuid: "75404e021bf34883abbce8a9c6470523" instance_seqno: 1781943521562433 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:18:41.960956 15861 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:18:41.980810 15854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } attempt: 3
Jun 20 08:18:41 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.189@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:18:41.989701 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.981301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.24.249.188:45907)
0620 08:18:41.981369 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:41.981371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:41.981383 (+ 12us) server_negotiation.cc:407] Connection header received
0620 08:18:41.981414 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:41.981415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:41.981429 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:41.981516 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:41.981938 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.982751 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.983941 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.984256 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.984332 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.986835 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:41.986837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:41.986837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:41.986845 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:41.988471 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.988861 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.988862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.988862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.988921 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:41.989109 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:41.989110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:41.989111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:41.989307 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:41.989376 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:41.989523 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 08:18:41.989603 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 08:18:41.989727 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:41.981212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44140)
0620 08:18:41.981272 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:41.981286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:41.981339 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:41.981533 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:41.981534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:41.981749 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:41.981808 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.981810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.982871 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.982872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.983850 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:41.983853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:41.984338 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:41.984339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:41.984404 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:41.984938 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:41.984943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:41.986734 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:41.988550 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.988551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.988551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.988779 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.988987 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:41.988989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:41.988989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:41.989051 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:41.989375 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:41.989376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:41.989419 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 08:18:41.989486 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 08:18:41.989584 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 08:18:41.991911 15854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: Bootstrap starting.
I20260620 08:18:41.992656 15854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:41.992974 15854 log.cc:824] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:41.994014 15854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: No bootstrap required, opened a new log
I20260620 08:18:41.996109 15854 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:41.996272 15854 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6a4b76874042549af2904f1b0c6f8e, State: Initialized, Role: FOLLOWER
I20260620 08:18:41.996349 15854 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:41.996696 15854 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:41.996935 15949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:41.997061 15949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 08:18:41.998646 15960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 08:18:41.998767 15960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 08:18:41.998862 15786 master_runner.cc:186] Error getting master registration for 127.24.249.190:34615: OK, instance_id { permanent_uuid: "75404e021bf34883abbce8a9c6470523" instance_seqno: 1781943521562433 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 08:18:42.000695 15786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:18:42.007454 15779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } attempt: 3
Jun 20 08:18:42 dist-test-slave-65w8 krb5kdc[15698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781943521, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.249.190@KRBTEST.COM for kudu/127.24.249.188@KRBTEST.COM
I20260620 08:18:42.016589 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:42.008089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44150 (local address 127.24.249.188:45907)
0620 08:18:42.008137 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.008139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.008163 (+ 24us) server_negotiation.cc:407] Connection header received
0620 08:18:42.008195 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.008196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.008211 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.008301 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.008674 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.009566 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.010488 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.010824 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.010902 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:42.013306 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:42.013308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:42.013308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:42.013318 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:42.015277 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.015702 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.015703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.015704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.015760 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.015960 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.015962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.015962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.016126 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:42.016207 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:42.016389 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 08:18:42.016488 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:18:42.016589 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:42.007833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.249.188:45907 (local address 127.0.0.1:44150)
0620 08:18:42.007903 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 08:18:42.007950 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 08:18:42.007999 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 08:18:42.008330 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 08:18:42.008332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 08:18:42.008517 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 08:18:42.008564 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:42.008566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.009656 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.009657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:42.010386 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 08:18:42.010388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.010918 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.010919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 08:18:42.011002 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:42.011532 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 08:18:42.011539 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 08:18:42.013207 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 08:18:42.015383 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:42.015385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:42.015386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:42.015621 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:42.015842 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 08:18:42.015843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 08:18:42.015843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 08:18:42.015903 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 08:18:42.016212 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 08:18:42.016213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 08:18:42.016280 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 08:18:42.016353 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 08:18:42.016472 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 08:18:42.018821 15779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523: Bootstrap starting.
I20260620 08:18:42.019634 15779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523: Neither blocks nor log segments found. Creating new log.
I20260620 08:18:42.019973 15779 log.cc:824] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523: Log is configured to *not* fsync() on all Append() calls
I20260620 08:18:42.020890 15779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523: No bootstrap required, opened a new log
I20260620 08:18:42.022544 15779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:42.022696 15779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75404e021bf34883abbce8a9c6470523, State: Initialized, Role: FOLLOWER
I20260620 08:18:42.022790 15779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:42.023168 15753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 08:18:42.023103 15963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.023267 15753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:42.023391 15963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:42.024035 15753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:42.024326 15753 leader_election.cc:288] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [CANDIDATE]: Term 1 election: Requested vote from peers ec6a4b76874042549af2904f1b0c6f8e (127.24.249.189:38497), 13043fa9fd504f4fbe7b5f97db7b2f10 (127.24.249.188:45907)
I20260620 08:18:42.023138 15779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 08:18:42.024832 15828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75404e021bf34883abbce8a9c6470523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec6a4b76874042549af2904f1b0c6f8e"
I20260620 08:18:42.024998 15828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:42.026173 15828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75404e021bf34883abbce8a9c6470523 in term 1.
I20260620 08:18:42.027616 15730 leader_election.cc:302] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75404e021bf34883abbce8a9c6470523, ec6a4b76874042549af2904f1b0c6f8e; no voters:
I20260620 08:18:42.028107 15903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75404e021bf34883abbce8a9c6470523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10"
I20260620 08:18:42.028247 15903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [term 0 FOLLOWER]: Advancing to term 1
I20260620 08:18:42.029282 15903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75404e021bf34883abbce8a9c6470523 in term 1.
I20260620 08:18:42.031189 15711 master_runner.cc:263] No leader master found from master 75404e021bf34883abbce8a9c6470523
I20260620 08:18:42.031236 15711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 08:18:42.031344 15963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 08:18:42.031561 15963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [term 1 LEADER]: Becoming Leader. State: Replica: 75404e021bf34883abbce8a9c6470523, State: Running, Role: LEADER
I20260620 08:18:42.031698 15963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } }
I20260620 08:18:42.032286 15963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75404e021bf34883abbce8a9c6470523. Latest consensus state: current_term: 1 leader_uuid: "75404e021bf34883abbce8a9c6470523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.032441 15963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:42.032716 15984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 08:18:42.033360 15984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 08:18:42.037671 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:42.027409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.24.249.190:34615)
0620 08:18:42.027455 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.027458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.028456 (+ 998us) server_negotiation.cc:407] Connection header received
0620 08:18:42.028566 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.028566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.028579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.028611 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.029514 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.030318 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.031418 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.031721 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.031770 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:42.033624 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:42.033625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:42.033626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:42.033633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:42.036146 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.036566 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.036567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.036568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.036623 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.036851 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.036852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.036853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.036988 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:42.037065 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:42.037404 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 08:18:42.037513 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:18:42.045636 15932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:42.027803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.24.249.188:45907)
0620 08:18:42.027842 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.027844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.034963 (+ 7119us) server_negotiation.cc:407] Connection header received
0620 08:18:42.035077 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.035079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.035091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.035131 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.035844 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.036632 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.038172 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.038501 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.038550 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:42.042335 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:42.042337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:42.042337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:42.042345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:42.044165 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.044599 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.044600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.044601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.044667 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.044921 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.044922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.044922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.045076 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:42.045174 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:42.045412 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 08:18:42.045503 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 08:18:42.047043 15828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [term 1 FOLLOWER]: Refusing update from remote peer 75404e021bf34883abbce8a9c6470523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:42.047165 15903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [term 1 FOLLOWER]: Refusing update from remote peer 75404e021bf34883abbce8a9c6470523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 08:18:42.047477 15982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:42.049109 15935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75404e021bf34883abbce8a9c6470523. Latest consensus state: current_term: 1 leader_uuid: "75404e021bf34883abbce8a9c6470523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.049219 15935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:42.049750 15963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 08:18:42.051239 15949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75404e021bf34883abbce8a9c6470523. Latest consensus state: current_term: 1 leader_uuid: "75404e021bf34883abbce8a9c6470523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.051359 15949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:42.052940 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:42.028133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.24.249.189:38497)
0620 08:18:42.028175 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.028177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.037815 (+ 9638us) server_negotiation.cc:407] Connection header received
0620 08:18:42.038135 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.038137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.038150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.038205 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.040566 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.042060 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.043931 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:42.044325 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:42.044372 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:42.046037 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 08:18:42.046038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 08:18:42.046039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 08:18:42.046047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 08:18:42.049675 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.051946 (+ 2271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.051947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.051948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.052013 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 08:18:42.052244 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 08:18:42.052245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 08:18:42.052245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 08:18:42.052410 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 08:18:42.052523 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:42.052743 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 08:18:42.052841 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:42.056506 15963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75404e021bf34883abbce8a9c6470523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.056638 15963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:42.058178 15949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75404e021bf34883abbce8a9c6470523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.058279 15949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:42.058646 15986 mvcc.cc:205] Tried to move back new op lower bound from 7298840666301206528 to 7298840666243149824. Current Snapshot: MvccSnapshot[applied={T|T < 7298840666301206528}]
I20260620 08:18:42.058854 15989 mvcc.cc:205] Tried to move back new op lower bound from 7298840666301206528 to 7298840666243149824. Current Snapshot: MvccSnapshot[applied={T|T < 7298840666301206528}]
I20260620 08:18:42.060024 15982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75404e021bf34883abbce8a9c6470523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.060145 15982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523 [sys.catalog]: This master's current role is: LEADER
I20260620 08:18:42.060628 15935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75404e021bf34883abbce8a9c6470523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75404e021bf34883abbce8a9c6470523" member_type: VOTER last_known_addr { host: "127.24.249.190" port: 34615 } } peers { permanent_uuid: "ec6a4b76874042549af2904f1b0c6f8e" member_type: VOTER last_known_addr { host: "127.24.249.189" port: 38497 } } peers { permanent_uuid: "13043fa9fd504f4fbe7b5f97db7b2f10" member_type: VOTER last_known_addr { host: "127.24.249.188" port: 45907 } } }
I20260620 08:18:42.060742 15935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 08:18:42.060984 15987 mvcc.cc:205] Tried to move back new op lower bound from 7298840666301206528 to 7298840666243149824. Current Snapshot: MvccSnapshot[applied={T|T < 7298840666301206528}]
I20260620 08:18:42.061597 15984 catalog_manager.cc:1347] Generated new cluster ID: 8f3f5b71d0824a3397e83dbc3fdcbfd1
I20260620 08:18:42.061652 15984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 08:18:42.096272 15984 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 08:18:42.097069 15984 catalog_manager.cc:1504] Loading token signing keys...
I20260620 08:18:42.107933 15984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75404e021bf34883abbce8a9c6470523: Generated new TSK 0
I20260620 08:18:42.108269 15984 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 08:18:42.138319 15782 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.136453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.24.249.190:34615)
0620 08:18:42.136501 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.136503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.137521 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 08:18:42.137632 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.137634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.137646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.137676 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.138242 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51684: BlockingRecv error: recv got EOF from 127.0.0.1:51684 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 08:18:42.139305 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.136733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.24.249.189:38497)
0620 08:18:42.136772 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.136774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.138673 (+ 1899us) server_negotiation.cc:407] Connection header received
0620 08:18:42.138793 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.138794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.138806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.138861 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.139236 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47096: BlockingRecv error: recv got EOF from 127.0.0.1:47096 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 08:18:42.140143 15932 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.137128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.24.249.188:45907)
0620 08:18:42.137169 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.137171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.139630 (+ 2459us) server_negotiation.cc:407] Connection header received
0620 08:18:42.139737 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.139738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.139749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.139779 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.140085 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44170: BlockingRecv error: recv got EOF from 127.0.0.1:44170 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 08:18:42.142810 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.141960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.24.249.189:38497)
0620 08:18:42.141999 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.142001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.142200 (+ 199us) server_negotiation.cc:407] Connection header received
0620 08:18:42.142315 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.142317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.142351 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.142382 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.142759 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47102: BlockingRecv error: recv got EOF from 127.0.0.1:47102 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 08:18:42.145912 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.145281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.24.249.189:38497)
0620 08:18:42.145317 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.145319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.145398 (+ 79us) server_negotiation.cc:407] Connection header received
0620 08:18:42.145424 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.145425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.145435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.145492 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.145823 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47108: BlockingRecv error: recv got EOF from 127.0.0.1:47108 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 08:18:42.149606 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.148921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.24.249.189:38497)
0620 08:18:42.148960 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.148962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.149109 (+ 147us) server_negotiation.cc:407] Connection header received
0620 08:18:42.149174 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.149175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.149187 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.149244 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.149557 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47116: BlockingRecv error: recv got EOF from 127.0.0.1:47116 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 08:18:42.155485 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.154630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.24.249.189:38497)
0620 08:18:42.154669 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.154671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.154871 (+ 200us) server_negotiation.cc:407] Connection header received
0620 08:18:42.154906 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.154908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.154922 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.155005 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.155427 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47128: BlockingRecv error: recv got EOF from 127.0.0.1:47128 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 08:18:42.165334 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.164545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.24.249.189:38497)
0620 08:18:42.164590 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.164592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.164746 (+ 154us) server_negotiation.cc:407] Connection header received
0620 08:18:42.164810 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.164811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.164825 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.164887 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.165261 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47132: BlockingRecv error: recv got EOF from 127.0.0.1:47132 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 08:18:42.183414 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.182490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.24.249.189:38497)
0620 08:18:42.182539 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.182541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.182710 (+ 169us) server_negotiation.cc:407] Connection header received
0620 08:18:42.182832 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.182834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.182848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.182913 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.183352 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47146: BlockingRecv error: recv got EOF from 127.0.0.1:47146 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 08:18:42.217557 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.216625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.24.249.189:38497)
0620 08:18:42.216733 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.216735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.216885 (+ 150us) server_negotiation.cc:407] Connection header received
0620 08:18:42.216969 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.216970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.216981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.217033 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.217477 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47160: BlockingRecv error: recv got EOF from 127.0.0.1:47160 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 08:18:42.283660 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.282824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.24.249.189:38497)
0620 08:18:42.282903 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.282906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.283044 (+ 138us) server_negotiation.cc:407] Connection header received
0620 08:18:42.283119 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.283120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.283133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.283218 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.283592 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47166: BlockingRecv error: recv got EOF from 127.0.0.1:47166 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 08:18:42.413568 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.412923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.24.249.189:38497)
0620 08:18:42.412990 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.412992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.413001 (+ 9us) server_negotiation.cc:407] Connection header received
0620 08:18:42.413033 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.413033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.413043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.413089 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.413475 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47174: BlockingRecv error: recv got EOF from 127.0.0.1:47174 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 08:18:42.671478 15858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 08:18:42.670919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.24.249.189:38497)
0620 08:18:42.670969 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:42.670971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:42.670987 (+ 16us) server_negotiation.cc:407] Connection header received
0620 08:18:42.671066 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:42.671067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:42.671077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:42.671129 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 08:18:42.671430 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47180: BlockingRecv error: recv got EOF from 127.0.0.1:47180 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:18:42.958222 15946 catalog_manager.cc:1259] Loaded cluster ID: 8f3f5b71d0824a3397e83dbc3fdcbfd1
I20260620 08:18:42.958335 15946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: loading cluster ID for follower catalog manager: success
I20260620 08:18:42.959182 15946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: acquiring CA information for follower catalog manager: success
I20260620 08:18:42.959510 15946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13043fa9fd504f4fbe7b5f97db7b2f10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:18:42.999279 15960 catalog_manager.cc:1259] Loaded cluster ID: 8f3f5b71d0824a3397e83dbc3fdcbfd1
I20260620 08:18:42.999511 15960 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: loading cluster ID for follower catalog manager: success
I20260620 08:18:43.000329 15960 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: acquiring CA information for follower catalog manager: success
I20260620 08:18:43.000624 15960 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec6a4b76874042549af2904f1b0c6f8e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 08:18:43.187273 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:43.184682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.24.249.189:38497)
0620 08:18:43.184809 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:43.184812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:43.184826 (+ 14us) server_negotiation.cc:407] Connection header received
0620 08:18:43.184880 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:43.184881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:43.184934 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:43.185007 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:43.185202 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.185732 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.186526 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.186719 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.186772 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:43.186946 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:43.187052 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:43.187108 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:43.187185 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 08:18:43.187203 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260620 08:18:43.191934 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:43.189291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.24.249.188:45907)
0620 08:18:43.189409 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:43.189411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:43.189480 (+ 69us) server_negotiation.cc:407] Connection header received
0620 08:18:43.189589 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:43.189590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:43.189665 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:43.189753 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:43.189901 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.190383 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.191128 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.191344 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.191410 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:43.191584 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:43.191697 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:43.191781 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:43.191810 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 08:18:43.191830 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260620 08:18:43.197050 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:43.193914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.24.249.188:45907)
0620 08:18:43.193958 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:43.193961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:43.194200 (+ 239us) server_negotiation.cc:407] Connection header received
0620 08:18:43.194290 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:43.194291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:43.194302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:43.194355 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:43.194500 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.195141 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.196201 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.196501 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.196572 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:43.196778 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:43.196813 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:43.196843 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:43.196961 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 08:18:43.196983 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 08:18:43.197121 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:43.193628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.24.249.189:38497)
0620 08:18:43.193687 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:43.193689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:43.193772 (+ 83us) server_negotiation.cc:407] Connection header received
0620 08:18:43.193838 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:43.193839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:43.193849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:43.193922 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:43.194141 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.194917 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.195869 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.196136 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.196227 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:43.196799 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:43.196840 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:43.196909 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:43.196976 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 08:18:43.196999 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 08:18:43.197574 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 08:18:43.193683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.24.249.190:34615)
0620 08:18:43.193829 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 08:18:43.193832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 08:18:43.194269 (+ 437us) server_negotiation.cc:407] Connection header received
0620 08:18:43.194321 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 08:18:43.194322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 08:18:43.194379 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 08:18:43.194454 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 08:18:43.194616 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.195301 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.196310 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 08:18:43.196660 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 08:18:43.196733 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 08:18:43.197002 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 08:18:43.197114 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 08:18:43.197187 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 08:18:43.197293 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 08:18:43.197490 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260620 08:18:43.203285 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15711
I20260620 08:18:43.209038 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15786
I20260620 08:18:43.214145 25574 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5EkY3n/build/release/bin/kudu with pid 15861
2026-06-20T08:18:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2951 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5842 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279550 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST